Search arXiv⌕ Search

arXiv subjects

Linfeng Zhang

Publications and source records attributed to Linfeng Zhang.

At least 271 records · Page 15Linked to original sources
arXiv2023

Revisiting Data Augmentation in Model Compression: An Empirical and Comprehensive Study

The excellent performance of deep neural networks is usually accompanied by a large number of parameters and computations, which have limited their usage on the resource-limited edge devices. To address this issue, abundant methods such as pruning, quantization and knowledge distillation have been proposed to compress neural networks and achieved significant breakthroughs. However, most of these compression methods focus on the architecture or the training method of neural networks but ignore the influence from data augmentation. In this paper, we revisit the usage of data augmentation in model compression and give a comprehensive study on the relation between model sizes and their optimal data augmentation policy. To sum up, we mainly have the following three observations: (A) Models in different sizes prefer data augmentation with different magnitudes. Hence, in iterative pruning, data augmentation with varying magnitudes leads to better performance than data augmentation with a consistent magnitude. (B) Data augmentation with a high magnitude may significantly improve the performance of large models but harm the performance of small models. Fortunately, small models can still benefit from strong data augmentations by firstly learning them with "additional parameters" and then discard these "additional parameters" during inference. (C) The prediction of a pre-trained large model can be utilized to measure the difficulty of data augmentation. Thus it can be utilized as a criterion to design better data augmentation policies. We hope this paper may promote more research on the usage of data augmentation in model compression.

cs.CV↗
arXiv2023

Thermal Conductivity of Water at Extreme Conditions

Measuring the thermal conductivity ($κ$) of water at extreme conditions is a challenging task and few experimental data are available. We predict $κ$ for temperatures and pressures relevant to the conditions of the Earth mantle, between 1,000 and 2,000 K and up to 22 GPa. We employ close to equilibrium molecular dynamics simulations and a deep neural network potential fitted to density functional theory data. We then interpret our results by computing the equation of state of water on a fine grid of points and using a simple model for $κ$. We find that the thermal conductivity is weakly dependent on temperature and monotonically increases with pressure with an approximate square-root behavior. In addition we show how the increase of $κ$ at high pressure, relative to ambient conditions, is related to the corresponding increase in the sound velocity. Although the relationships between the thermal conductivity, pressure and sound velocity established here are not rigorous, they are sufficiently accurate to allow for a robust estimate of the thermal conductivity of water in a broad range of temperatures and pressures, where experiments are still difficult to perform.

cond-mat.mtrl-sci↗
arXiv2023

CORSD: Class-Oriented Relational Self Distillation

Knowledge distillation conducts an effective model compression method while holding some limitations:(1) the feature based distillation methods only focus on distilling the feature map but are lack of transferring the relation of data examples; (2) the relational distillation methods are either limited to the handcrafted functions for relation extraction, such as L2 norm, or weak in inter- and intra- class relation modeling. Besides, the feature divergence of heterogeneous teacher-student architectures may lead to inaccurate relational knowledge transferring. In this work, we propose a novel training framework named Class-Oriented Relational Self Distillation (CORSD) to address the limitations. The trainable relation networks are designed to extract relation of structured data input, and they enable the whole model to better classify samples by transferring the relational knowledge from the deepest layer of the model to shallow layers. Besides, auxiliary classifiers are proposed to make relation networks capture class-oriented relation that benefits classification task. Experiments demonstrate that CORSD achieves remarkable improvements. Compared to baseline, 3.8%, 1.5% and 4.5% averaged accuracy boost can be observed on CIFAR100, ImageNet and CUB-200-2011, respectively.

cs.CV↗
arXiv2023

Imperfectly coordinated water molecules pave the way for homogeneous ice nucleation

Water freezing is ubiquitous on Earth, affecting many areas from biology to climate science and aviation technology. Probing the atomic structure in the homogeneous ice nucleation process from scratch is of great value but still experimentally unachievable. Theoretical simulations have found that ice originates from the low-mobile region with increasing abundance and persistence of tetrahedrally coordinated water molecules. However, a detailed microscopic picture of how the disordered hydrogen-bond network rearranges itself into an ordered network is still unclear. In this work, we use a deep neural network (DNN) model to "learn" the interatomic potential energy from quantum mechanical data, thereby allowing for large-scale and long molecular dynamics (MD) simulations with ab initio accuracy. The nucleation mechanism and dynamics at atomic resolution, represented by a total of 36 $μ$s-long MD trajectories, are deeply affected by the structural and dynamical heterogeneity in supercooled water. We find that imperfectly coordinated (IC) water molecules with high mobility pave the way for hydrogen-bond network rearrangement, leading to the growth or shrinkage of the ice nucleus. The hydrogen-bond network formed by perfectly coordinated (PC) molecules stabilizes the nucleus, thus preventing it from vanishing and growing. Consequently, ice is born through competition and cooperation between IC and PC molecules. We anticipate that our picture of the microscopic mechanism of ice nucleation will provide new insights into many properties of water and other relevant materials.

cond-mat.mtrl-sci↗
arXiv2023

Uni-QSAR: an Auto-ML Tool for Molecular Property Prediction

Recently deep learning based quantitative structure-activity relationship (QSAR) models has shown surpassing performance than traditional methods for property prediction tasks in drug discovery. However, most DL based QSAR models are restricted to limited labeled data to achieve better performance, and also are sensitive to model scale and hyper-parameters. In this paper, we propose Uni-QSAR, a powerful Auto-ML tool for molecule property prediction tasks. Uni-QSAR combines molecular representation learning (MRL) of 1D sequential tokens, 2D topology graphs, and 3D conformers with pretraining models to leverage rich representation from large-scale unlabeled data. Without any manual fine-tuning or model selection, Uni-QSAR outperforms SOTA in 21/22 tasks of the Therapeutic Data Commons (TDC) benchmark under designed parallel workflow, with an average performance improvement of 6.09\%. Furthermore, we demonstrate the practical usefulness of Uni-QSAR in drug discovery domains.

q-bio.BM↗
arXiv2023

Hybrid Auxiliary Field Quantum Monte Carlo for Molecular Systems

We propose a quantum Monte Carlo approach to solve the ground state many-body Schrodinger equation for the electronic ground state. The method combines optimization from variational Monte Carlo and propagation from auxiliary field quantum Monte Carlo, in a way that significantly alleviates the sign problem. In application to molecular systems, we obtain highly accurate results for configurations dominated by either dynamic or static electronic correlation.

physics.comp-ph↗
arXiv2023

DeePMD-kit v2: A software package for Deep Potential models

DeePMD-kit is a powerful open-source software package that facilitates molecular dynamics simulations using machine learning potentials (MLP) known as Deep Potential (DP) models. This package, which was released in 2017, has been widely used in the fields of physics, chemistry, biology, and material science for studying atomistic systems. The current version of DeePMD-kit offers numerous advanced features such as DeepPot-SE, attention-based and hybrid descriptors, the ability to fit tensile properties, type embedding, model deviation, Deep Potential - Range Correction (DPRc), Deep Potential Long Range (DPLR), GPU support for customized operators, model compression, non-von Neumann molecular dynamics (NVNMD), and improved usability, including documentation, compiled binary packages, graphical user interfaces (GUI), and application programming interfaces (API). This article presents an overview of the current major version of the DeePMD-kit package, highlighting its features and technical details. Additionally, the article benchmarks the accuracy and efficiency of different models and discusses ongoing developments.

physics.chem-ph↗
arXiv2023

Modeling the High-Pressure Solid and Liquid Phases of Tin from Deep Potentials with ab initio Accuracy

Constructing an accurate atomistic model for the high-pressure phases of tin (Sn) is challenging because properties of Sn are sensitive to pressures. We develop machine-learning-based deep potentials for Sn with pressures ranging from 0 to 50 GPa and temperatures ranging from 0 to 2000 K. In particular, we find the deep potential, which is obtained by training the ab initio data from density functional theory calculations with the state-of-the-art SCAN exchange-correlation functional, is suitable to characterize high-pressure phases of Sn. We systematically validate several structural and elastic properties of the α (diamond structure), β, bct, and bcc structures of Sn, as well as the structural and dynamic properties of liquid Sn. The thermodynamics integration method is further utilized to compute the free energies of the α, β, bct, and liquid phases, from which the deep potential successfully predicts the phase diagram of Sn including the existence of the triple-point that qualitatively agrees with the experiment.

cond-mat.mtrl-sci↗
arXiv2023

Modular development of deep potential for complex solid solutions

The multicomponent oxide solid solution is a versatile platform to tune the delicate balance between competing spin, charge, orbital, and lattice degrees of freedom for materials design and discovery. The development of compositionally complex oxides with superior functional properties has been largely empirical and serendipitous, in part due to the exceedingly complex chemistry and structure of solid solutions that span a range of length scales. The classical molecular dynamics (MD), as a powerful statistical method to investigate materials properties over large spatial and temporal scales, often plays a secondary role in computer-aided materials discovery because of the limited availability and accuracy of classical force fields. Here, we introduce the strategy of ``modular developing deep potential" (ModDP) that enables a systematic development and improvement of deep neural network-based model potential, termed as deep potential, for complex solid solutions with minimum human intervention. The converged training database associated with an end-member material is treated as an independent module and is reused to train the deep potential of solid solutions via a concurrent learning procedure. We apply ModDP to obtain classical force fields of two technologically important solid solutions, Pb$_x$Sr$_{1-x}$TiO$_3$ and Hf$_x$Zr$_{1-x}$O$_2$. For both materials systems, a single model potential is capable of predicting various properties of solid solutions including temperature-driven and composition-driven phase transitions over a wide range of compositions. In particular, the deep potential of Pb$_x$Sr$_{1-x}$TiO$_3$ reproduces a few known topological textures such as polar vortex lattice and electric dipole waves in PbTiO$_3$/SrTiO$_3$ superlattices, paving the way for MD investigations on the dynamics of topological structures in response to external stimuli.

cond-mat.mtrl-sci↗
arXiv2023

Entangled X-ray Photon Pair Generation by Free Electron Lasers

Einstein, Podolsky and Rosen's prediction on incompleteness of quantum mechanics was overturned by experimental tests on Bell's inequality that confirmed the existence of quantum entanglement. In X-ray optics, entangled photon pairs can be generated by X-ray parametric down conversion (XPDC), which has certain wavelength window. Meanwhile, free electron laser (FEL) has successfully lased at X-ray frequencies recently. However, FEL is usually seen as a classical light source, and its quantum effects are considered minor corrections to the classical theory. Here we investigate entangled X-ray photon pair emissions in FEL. We establish a theory for coherently amplified entangled photon pair emission from microbunched electron pulses in the undulator. We numerically demonstrate the properties of entangled emission, and provide a scheme to generate highly entangled X-ray photon pairs, which is of great importance in X-ray quantum optics. Our work shows a unique advantage of FELs over synchrotrons in entangled X-ray photon pair generation.

quant-ph↗
arXiv2023

Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?

The success of deep learning heavily relies on large-scale data with comprehensive labels, which is more expensive and time-consuming to fetch in 3D compared to 2D images or natural languages. This promotes the potential of utilizing models pretrained with data more than 3D as teachers for cross-modal knowledge transferring. In this paper, we revisit masked modeling in a unified fashion of knowledge distillation, and we show that foundational Transformers pretrained with 2D images or natural languages can help self-supervised 3D representation learning through training Autoencoders as Cross-Modal Teachers (ACT). The pretrained Transformers are transferred as cross-modal 3D teachers using discrete variational autoencoding self-supervision, during which the Transformers are frozen with prompt tuning for better knowledge inheritance. The latent features encoded by the 3D teachers are used as the target of masked point modeling, wherein the dark knowledge is distilled to the 3D Transformer students as foundational geometry understanding. Our ACT pretrained 3D learner achieves state-of-the-art generalization capacity across various downstream benchmarks, e.g., 88.21% overall accuracy on ScanObjectNN. Codes have been released at https://github.com/RunpeiDong/ACT.

cs.CV↗
arXiv2022

Atomistic modelling of all dislocations and twins in HCP and BCC Ti

Ti exhibits complex plastic deformation controlled by active dislocation and twinning systems. Understandings on dislocation cores and twin interfaces are currently not complete or quantitative, despite extensive experimental and simulation studies. Here, we determine all the core and twin interface properties in both HCP and BCC Ti using a Deep Potential (DP) and DFT. We determine the core structures, critical resolved shear stresses and mobilities of , , dislocations in HCP and <111>/2 dislocations in BCC Ti. The slip consists of slow core migration on pyramidal-I planes and fast migration on prism-planes, and is kinetically limited by cross-slips among them. This behaviour is consistent with "locking-unlocking" phenomena in TEM and is likely an intrinsic property. Large-scale DFT calculations provide a peek at the screw core and glide behaviour, which is further quantified using DP-Ti. The screw is unstable on pyramidal-II planes. The mixed is nearly sessile on pyramidal-I planes, consistent with observations of long dislocations in this orientation. The edge and mixed are unstable against a pyramidal-to-basal (PB) transition and become sessile at high temperatures, corroborate the difficulties in -axis compression of Ti. Finally, in BCC Ti, the <111>/2 screw has a degenerate core with average glide on {112} planes; the <111>/2 edge and mixed dislocations have non-dissociated cores on {110} planes. This work paints a self-consistent, complete picture on all dislocations in Ti, rationalises previous experimental observations and points to future HRTEM examinations of unusual dislocations such as the mixed and PB transformed cores.

cond-mat.mtrl-sci↗
arXiv2022

Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detection

Detecting 3D objects from multi-view images is a fundamental problem in 3D computer vision. Recently, significant breakthrough has been made in multi-view 3D detection tasks. However, the unprecedented detection performance of these vision BEV (bird's-eye-view) detection models is accompanied with enormous parameters and computation, which make them unaffordable on edge devices. To address this problem, in this paper, we propose a structured knowledge distillation framework, aiming to improve the efficiency of modern vision-only BEV detection models. The proposed framework mainly includes: (a) spatial-temporal distillation which distills teacher knowledge of information fusion from different timestamps and views, (b) BEV response distillation which distills teacher response to different pillars, and (c) weight-inheriting which solves the problem of inconsistent inputs between students and teacher in modern transformer architectures. Experimental results show that our method leads to an average improvement of 2.16 mAP and 2.27 NDS on the nuScenes benchmark, outperforming multiple baselines by a large margin.

cs.CV↗
arXiv2022

Machine learning assisted coarse-grained molecular dynamics modeling of meso-scale interfacial fluids

A hallmark of meso-scale interfacial fluids is the multi-faceted, scale-dependent interfacial energy, which often manifests different characteristics across the molecular and continuum scale. The multi-scale nature imposes a challenge to construct reliable coarse-grained (CG) models, where the CG potential function needs to faithfully encode the many-body interactions arising from the unresolved atomistic interactions and account for the heterogeneous density distributions across the interface. We construct the CG models of both single- and two-component of polymeric fluid systems based on the recently developed deep coarse-grained potential (DeePCG) scheme, where each polymer molecule is modeled as a CG particle. By only using the training samples of the instantaneous force under the thermal equilibrium state, the constructed CG models can accurately reproduce both the probability density function of the void formation in bulk and the spectrum of the capillary wave across the fluid interface. More importantly, the CG models accurately predict the volume-to-area scaling transition for the apolar solvation energy, illustrating the effectiveness to probe the meso-scale collective behaviors encoded with molecular-level fidelity.

physics.comp-ph↗
arXiv2022

Classical and Machine Learning Interatomic Potentials for BCC Vanadium

BCC transition metals (TMs) exhibit complex temperature and strain-rate dependent plastic deformation behaviour controlled by individual crystal lattice defects. Classical empirical and semi-empirical interatomic potentials have limited capability in modelling defect properties such as the screw dislocation core structures and Peierls barriers in the BCC structure. Machine learning (ML) potentials, trained on DFT-based datasets, have shown some successes in reproducing dislocation core properties. However, in group VB TMs, the most widely-used DFT functionals produce erroneous shear moduli C44 which are undesirably transferred to machine-learning interatomic potentials, leaving current ML approaches unsuitable for this important class of metals and alloys. Here, we develop two interatomic potentials for BCC vanadium (V) based on (i) an extension of the partial electron density and screening parameter in the classical semi-empirical modified embedded-atom method (XMEAM-V) and (ii) a recent hybrid descriptor in the ML Deep Potential framework (DP-HYB-V). We describe distinct features in these two disparate approaches, including their dataset generation, training procedure, weakness and strength in modelling lattice and defect properties in BCC V. Both XMEAM-V and DP-HYB-V reproduce a broad range of defect properties relevant to plastic deformation and fracture. In particular, XMEAM-V reproduces nearly all mechanical and thermodynamic properties at DFT accuracies and with C44 near experimental value. XMEAM-V also naturally exhibits the anomalous slip at 77 K widely observed in group VB and VIB TMs and outperforms all existing, publically available interatomic potentials for V. The XMEAM thus provides a practical path to developing accurate and efficient interatomic potentials for nonmagnetic BCC TMs and possibly multi-principal element TM alloys.

cond-mat.mtrl-sci↗
arXiv2022

DeePKS+ABACUS as a Bridge between Expensive Quantum Mechanical Models and Machine Learning Potentials

Recently, the development of machine learning (ML) potentials has made it possible to perform large-scale and long-time molecular simulations with the accuracy of quantum mechanical (QM) models. However, for high-level QM methods, such as density functional theory (DFT) at the meta-GGA level and/or with exact exchange, quantum Monte Carlo, etc., generating a sufficient amount of data for training a ML potential has remained computationally challenging due to their high cost. In this work, we demonstrate that this issue can be largely alleviated with Deep Kohn-Sham (DeePKS), a ML-based DFT model. DeePKS employs a computationally efficient neural network-based functional model to construct a correction term added upon a cheap DFT model. Upon training, DeePKS offers closely-matched energies and forces compared with high-level QM method, but the number of training data required is orders of magnitude less than that required for training a reliable ML potential. As such, DeePKS can serve as a bridge between expensive QM models and ML potentials: one can generate a decent amount of high-accuracy QM data to train a DeePKS model, and then use the DeePKS model to label a much larger amount of configurations to train a ML potential. This scheme for periodic systems is implemented in a DFT package ABACUS, which is open-source and ready for use in various applications.

physics.chem-ph↗
arXiv2022

DP Compress: a Model Compression Scheme for Generating Efficient Deep Potential Models

Machine-learning-based interatomic potential energy surface (PES) models are revolutionizing the field of molecular modeling. However, although much faster than electronic structure schemes, these models suffer from costly computations via deep neural networks to predict the energy and atomic forces, resulting in lower running efficiency as compared to the typical empirical force fields. Herein, we report a model compression scheme for boosting the performance of the Deep Potential (DP) model, a deep learning based PES model. This scheme, we call DP Compress, is an efficient post-processing step after the training of DP models (DP Train). DP Compress combines several DP-specific compression techniques, which typically speed up DP-based molecular dynamics simulations by an order of magnitude faster, and consume an order of magnitude less memory. We demonstrate that DP Compress is sufficiently accurate by testing a variety of physical properties of Cu, H2O, and Al-Cu-Mg systems. DP Compress applies to both CPU and GPU machines and is publicly available online.

physics.comp-ph↗
arXiv2022

Contrastive Deep Supervision

The success of deep learning is usually accompanied by the growth in neural network depth. However, the traditional training method only supervises the neural network at its last layer and propagates the supervision layer-by-layer, which leads to hardship in optimizing the intermediate layers. Recently, deep supervision has been proposed to add auxiliary classifiers to the intermediate layers of deep neural networks. By optimizing these auxiliary classifiers with the supervised task loss, the supervision can be applied to the shallow layers directly. However, deep supervision conflicts with the well-known observation that the shallow layers learn low-level features instead of task-biased high-level semantic features. To address this issue, this paper proposes a novel training framework named Contrastive Deep Supervision, which supervises the intermediate layers with augmentation-based contrastive learning. Experimental results on nine popular datasets with eleven models demonstrate its effects on general image classification, fine-grained image classification and object detection in supervised learning, semi-supervised learning and knowledge distillation. Codes have been released in Github.

cs.CV↗