arXiv subjects

Jianfeng Lu

Publications and source records attributed to Jianfeng Lu.

At least 199 records · Page 11Linked to original sources

arXiv2020

LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning

While pre-training and fine-tuning, e.g., BERT~\citep{devlin2018bert}, GPT-2~\citep{radford2019language}, have achieved great success in language understanding and generation tasks, the pre-trained models are usually too big for online deployment in terms of both memory cost and inference speed, which hinders them from practical online usage. In this paper, we propose LightPAFF, a Lightweight Pre-training And Fine-tuning Framework that leverages two-stage knowledge distillation to transfer knowledge from a big teacher model to a lightweight student model in both pre-training and fine-tuning stages. In this way the lightweight model can achieve similar accuracy as the big teacher model, but with much fewer parameters and thus faster online inference speed. LightPAFF can support different pre-training methods (such as BERT, GPT-2 and MASS~\citep{song2019mass}) and be applied to many downstream tasks. Experiments on three language understanding tasks, three language modeling tasks and three sequence to sequence generation tasks demonstrate that while achieving similar accuracy with the big BERT, GPT-2 and MASS models, LightPAFF reduces the model size by nearly 5x and improves online inference speed by 5x-7x.

cs.CL↗

arXiv2020

Synchronization of Kuramoto Oscillators in Dense Networks

We study synchronization properties of systems of Kuramoto oscillators. The problem can also be understood as a question about the properties of an energy landscape created by a graph. More formally, let $G=(V,E)$ be a connected graph and $(a_{ij})_{i,j=1}^{n}$ denotes its adjacency matrix. Let the function $f:\mathbb{T}^n \rightarrow \mathbb{R}$ be given by $$ f(θ_1, \dots, θ_n) = \sum_{i,j=1}^{n}{ a_{ij} \cos{(θ_i - θ_j)}}.$$ This function has a global maximum when $θ_i = θ$ for all $1\leq i \leq n$. It is known that if every vertex is connected to at least $μ(n-1)$ other vertices for $μ$ sufficiently large, then every local maximum is global. Taylor proved this for $μ\geq 0.9395$ and Ling, Xu \& Bandeira improved this to $μ\geq 0.7929$. We give a slight improvement to $μ\geq 0.7889$. Townsend, Stillman \& Strogatz suggested that the critical value might be $μ_c = 0.75$.

math.OC↗

arXiv2020

Bloch dynamics with second order Berry phase correction

We derive the semiclassical Bloch dynamics with the second-order Berry phase correction in the presence of the slow-varying scalar potential as perturbation. Our mathematical derivation is based on a two-scale WKB asymptotic analysis. For a uniform external electric field, the bi-characteristics system after a positional shift introduced by Berry connections agrees with the recent result in previous works. Moreover, for the case with a linear external electric field, we show that the extra terms arising in the bi-characteristics system after the positional shift are also gauge independent.

math.AP↗

arXiv2020

Convergence of Stochastic-extended Lagrangian molecular dynamics method for polarizable force field simulation

Extended Lagrangian molecular dynamics (XLMD) is a general method for performing molecular dynamics simulations using quantum and classical many-body potentials. Recently several new XLMD schemes have been proposed and tested on several classes of many-body polarization models such as induced dipoles or Drude charges, by creating an auxiliary set of these same degrees of freedom that are reversibly integrated through time. This gives rise to a singularly perturbed Hamiltonian system that provides a good approximation to the time evolution of the real mutual polarization field. To further improve upon the accuracy of the XLMD dynamics, and to potentially extend it to other many-body potentials, we introduce a stochastic modification which leads to a set of singularly perturbed Langevin equations with degenerate noise. We prove that the resulting Stochastic-XLMD converges to the accurate dynamics, and the convergence rate is both optimal and is independent of the accuracy of the initial polarization field. We carefully study the scaling of the damping factor and numerical noise for efficient numerical simulation for Stochastic-XLMD, and we demonstrate the effectiveness of the method for model polarizable force field systems.

math.NA↗

arXiv2020

Continuum limit and preconditioned Langevin sampling of the path integral molecular dynamics

We investigate the continuum limit that the number of beads goes to infinity in the ring polymer representation of thermal averages. Studying the continuum limit of the trajectory sampling equation sheds light on possible preconditioning techniques for sampling ring polymer configurations with large number of beads. We propose two preconditioned Langevin sampling dynamics, which are shown to have improved stability and sampling accuracy. We present a careful mode analysis of the preconditioned dynamics and show their connections to the normal mode, the staging coordinate and the Matsubara mode representation for ring polymers. In the case where the potential is quadratic, we show that the continuum limit of the preconditioned mass modified Langevin dynamics converges to its equilibrium exponentially fast, which suggests that the finite-dimensional counterpart has a dimension-independent convergence rate. In addition, the preconditioning techniques can be naturally applied to the multi-level quantum systems in the nonadiabatic regime, which are compatible with various numerical approaches.

physics.chem-ph↗

arXiv2020

Random Sampling and Efficient Algorithms for Multiscale PDEs

We describe a numerical framework that uses random sampling to efficiently capture low-rank local solution spaces of multiscale PDE problems arising in domain decomposition. In contrast to existing techniques, our method does not rely on detailed analytical understanding of specific multiscale PDEs, in particular, their asymptotic limits. We present the application of the framework on two examples --- a linear kinetic equation and an elliptic equation with rough media. On these two examples, this framework achieves the asymptotic preserving property for the kinetic equations and numerical homogenization for the elliptic equations.

math.NA↗

arXiv2020

Non-Convex Planar Harmonic Maps

We formulate a novel characterization of a family of invertible maps between two-dimensional domains. Our work follows two classic results: The Radó-Kneser-Choquet (RKC) theorem, which establishes the invertibility of harmonic maps into a convex planer domain; and Tutte's embedding theorem for planar graphs - RKC's discrete counterpart - which proves the invertibility of piecewise linear maps of triangulated domains satisfying a discrete-harmonic principle, into a convex planar polygon. In both theorems, the convexity of the target domain is essential for ensuring invertibility. We extend these characterizations, in both the continuous and discrete cases, by replacing convexity with a less restrictive condition. In the continuous case, Alessandrini and Nesi provide a characterization of invertible harmonic maps into non-convex domains with a smooth boundary by adding additional conditions on orientation preservation along the boundary. We extend their results by defining a condition on the normal derivatives along the boundary, which we call the cone condition; this condition is tractable and geometrically intuitive, encoding a weak notion of local invertibility. The cone condition enables us to extend Alessandrini and Nesi to the case of harmonic maps into non-convex domains with a piecewise-smooth boundary. In the discrete case, we use an analog of the cone condition to characterize invertible discrete-harmonic piecewise-linear maps of triangulations. This gives an analog of our continuous results and characterizes invertible discrete-harmonic maps in terms of the orientation of triangles incident on the boundary.

math.AP↗

arXiv2019

Analysis of a continuum theory for broken bond crystal surface models with evaporation and deposition effects

We study a $4$th order PDE with a $2$nd order correction that is a modification of one suggested by Peter Smereka using the tools of Krug et al (Z. Phys. B., 1994). First, we provide a non-rigorous derivation of the full model derived with a quadratic interaction potential using the rates and methods of the last author with Jonathan Weare (PRE, 2013). Then, we prove the existence of a global in time weak solution by regularizing the equation in a way that allows us to apply the tools of Bernis-Friedman (JDE, 1990). The methods developed here can be applied to a large number of $4$th order degenerate models. In an appendix, we also discuss the global smooth solution with small data in the Weiner algebra framework following recent developments using tools of the second author with Robert Strain (IFB, 2018).

math.AP↗

arXiv2019

Part-based Multi-stream Model for Vehicle Searching

Due to the enormous requirement in public security and intelligent transportation system, searching an identical vehicle has become more and more important. Current studies usually treat vehicle as an integral object and then train a distance metric to measure the similarity among vehicles. However, these raw images may be exactly similar to ones with different identification and include some pixels in background that may disturb the distance metric learning. In this paper, we propose a novel and useful method to segment an original vehicle image into several discriminative foreground parts, and these parts consist of some fine grained regions that are named discriminative patches. After that, these parts combined with the raw image are fed into the proposed deep learning network. We can easily measure the similarity of two vehicle images by computing the Euclidean distance of the features from FC layer. Two main contributions of this paper are as follows. Firstly, a method is proposed to estimate if a patch in a raw vehicle image is discriminative or not. Secondly, a new Part-based Multi-Stream Model (PMSM) is designed and optimized for vehicle retrieval and re-identification tasks. We evaluate the proposed method on the VehicleID dataset, and the experimental results show that our method can outperform the baseline.

cs.CV↗

arXiv2019

Tensorization of the strong data processing inequality for quantum chi-square divergences

It is well-known that any quantum channel $\mathcal{E}$ satisfies the data processing inequality (DPI), with respect to various divergences, e.g., quantum $χ^2_κ$divergences and quantum relative entropy. More specifically, the data processing inequality states that the divergence between two arbitrary quantum states $ρ$ and $σ$ does not increase under the action of any quantum channel $\mathcal{E}$. For a fixed channel $\mathcal{E}$ and a state $σ$, the divergence between output states $\mathcal{E}(ρ)$ and $\mathcal{E}(σ)$ might be strictly smaller than the divergence between input states $ρ$ and $σ$, which is characterized by the strong data processing inequality (SDPI). Among various input states $ρ$, the largest value of the rate of contraction is known as the SDPI constant. An important and widely studied property for classical channels is that SDPI constants tensorize. In this paper, we extend the tensorization property to the quantum regime: we establish the tensorization of SDPIs for the quantum $χ^2_{κ_{1/2}}$ divergence for arbitrary quantum channels and also for a family of $χ^2_κ$ divergences (with $κ\ge κ_{1/2}$) for arbitrary quantum-classical channels.

quant-ph↗

arXiv2019

Stochastic modified equations for the asynchronous stochastic gradient descent

We propose a stochastic modified equations (SME) for modeling the asynchronous stochastic gradient descent (ASGD) algorithms. The resulting SME of Langevin type extracts more information about the ASGD dynamics and elucidates the relationship between different types of stochastic gradient algorithms. We show the convergence of ASGD to the SME in the continuous time limit, as well as the SME's precise prediction to the trajectories of ASGD with various forcing terms. As an application of the SME, we propose an optimal mini-batching strategy for ASGD via solving the optimal control problem of the associated SME.

stat.ML↗

arXiv2019

Discontinuous Hamiltonian Monte Carlo for discrete parameters and discontinuous likelihoods

Hamiltonian Monte Carlo has emerged as a standard tool for posterior computation. In this article, we present an extension that can efficiently explore target distributions with discontinuous densities. Our extension in particular enables efficient sampling from ordinal parameters though embedding of probability mass functions into continuous spaces. We motivate our approach through a theory of discontinuous Hamiltonian dynamics and develop a corresponding numerical solver. The proposed solver is the first of its kind, with a remarkable ability to exactly preserve the Hamiltonian. We apply our algorithm to challenging posterior inference problems to demonstrate its wide applicability and competitive performance.

stat.CO↗

arXiv2019

Optimal Trapping of Brownian Motion: A Nonlinear Analogue of the Torsion Function

We study the problem of maximizing the expected lifetime of drift diffusion in a bounded domain. More formally, we consider the PDE \[ - Δu + b(x) \cdot \nabla u = 1 \qquad \mbox{in}~Ω\] subject to Dirichlet boundary conditions for $\|b\|_{L^{\infty}}$ fixed. We show that, in any given $C^2-$domain $Ω$, the vector field maximizing the expected lifetime is (nonlinearly) coupled to the solution and satisfies $b = -\|b\|_{L^{\infty}} \nabla u/ |\nabla u|$ which reduces the problem to the study of the nonlinear PDE \[ -Δu - b \cdot \left| \nabla u \right| = 1,\] where $b = \|b\|_{L^{\infty}}$ is a constant. We believe that this PDE is a natural and interesting nonlinear analogue of the torsion function. We prove that, for fixed volume, $\| \nabla u\|_{L^1}$ and $\|Δu\|_{L^1}$ are maximized if $Ω$ is the ball (the ball is also known to maximize $\|u\|_{L^p}$ for $p \geq 1$ from a result of Hamel \& Russ).

math.AP↗

arXiv2019

Computing edge states without hard truncation

We present a numerical method which accurately computes the discrete spectrum and associated bound states of Hamiltonians which model electronic "edge" states localized at boundaries of one and two-dimensional crystalline materials. The problem is non-trivial since arbitrarily large finite "hard" truncations of the Hamiltonian in the infinite bulk direction tend to produce spurious bound states partially supported at the truncation. Our method, which overcomes this difficulty, is to compute the Green's function of the Hamiltonian by imposing an appropriate boundary condition in the bulk direction; then, the spectral data is recovered via Riesz projection. We demonstrate our method's effectiveness by studies of edge states at a graphene zig-zag edge in the presence of defects modeled both by a discrete tight-binding model and a continuum PDE model under finite difference discretization. Our method may also be used to study states localized at domain wall-type edges in one and two-dimensional materials where the edge Hamiltonian is infinite in both directions; we demonstrate this for the case of a tight-binding model of distinct honeycomb structures joined along a zig-zag edge.

physics.comp-ph↗

arXiv2019

Fisher information regularization schemes for Wasserstein gradient flows

We propose a variational scheme for computing Wasserstein gradient flows. The scheme builds upon the Jordan--Kinderlehrer--Otto framework with the Benamou-Brenier's dynamic formulation of the quadratic Wasserstein metric and adds a regularization by the Fisher information. This regularization can be derived in terms of energy splitting and is closely related to the Schr{ö}dinger bridge problem. It improves the convexity of the variational problem and automatically preserves the non-negativity of the solution. As a result, it allows us to apply sequential quadratic programming to solve the sub-optimization problem. We further save the computational cost by showing that no additional time interpolation is needed in the underlying dynamic formulation of the Wasserstein-2 metric, and therefore, the dimension of the problem is vastly reduced. Several numerical examples, including porous media equation, nonlinear Fokker-Planck equation, aggregation diffusion equation, and Derrida-Lebowitz-Speer-Spohn equation, are provided. These examples demonstrate the simplicity and stableness of the proposed scheme.

math.NA↗

arXiv2019

Dirac operators and domain walls

We study the eigenvalue problem for a one-dimensional Dirac operator with a spatially varying ``mass'' term. It is well-known that when the mass function has the form of a kink, or \emph{domain wall}, transitioning between strictly positive and strictly negative asymptotic mass, $\pmκ_\infty$, at $\pm\infty$, the Dirac operator has a simple eigenvalue of zero energy (geometric multiplicity equal to one) within a gap in the continuous spectrum, with corresponding \emph{zero mode}, an exponentially localized eigenfunction. We prove that when the mass function has the form of \emph{two} domain walls separated by a sufficiently large distance $2 δ$, the Dirac operator has two real simple eigenvalues of opposite sign and of order $e^{- 2 |κ_\infty| δ}$. The associated eigenfunctions are, up to $L^2$ error of order $e^{- 2 |κ_\infty| δ}$, linear combinations of shifted copies of the single domain wall zero mode. For the case of three domain walls, there are two non-zero simple eigenvalues as above and a simple eigenvalue at energy zero. Our methods are based on a Lyapunov-Schmidt reduction strategy and we outline their natural extension to the case of $n$ domain walls for which the minimal distance between domain walls is sufficiently large. The class of Dirac operators we consider controls the bifurcation of topologically protected ``edge states'' from Dirac points (linear band crossings) for classes of Schrödinger operators with domain-wall modulated periodic potentials in one and two space dimensions. The present results may be used to construct a rich class of defect modes in periodic structures modulated by multiple domain walls.

math.AP↗

arXiv2019

Efficient construction of tensor ring representations from sampling

In this paper we propose an efficient method to compress a high dimensional function into a tensor ring format, based on alternating least-squares (ALS). Since the function has size exponential in $d$ where $d$ is the number of dimensions, we propose efficient sampling scheme to obtain $O(d)$ important samples in order to learn the tensor ring. Furthermore, we devise an initialization method for ALS that allows fast convergence in practice. Numerical examples show that to approximate a function with similar accuracy, the tensor ring format provided by the proposed method has less parameters than tensor-train format and also better respects the structure of the original function.

math.NA↗

arXiv2019

MASS: Masked Sequence to Sequence Pre-training for Language Generation

Pre-training and fine-tuning, e.g., BERT, have achieved great success in language understanding by transferring knowledge from rich-resource pre-training task to the low/zero-resource downstream tasks. Inspired by the success of BERT, we propose MAsked Sequence to Sequence pre-training (MASS) for the encoder-decoder based language generation tasks. MASS adopts the encoder-decoder framework to reconstruct a sentence fragment given the remaining part of the sentence: its encoder takes a sentence with randomly masked fragment (several consecutive tokens) as input, and its decoder tries to predict this masked fragment. In this way, MASS can jointly train the encoder and decoder to develop the capability of representation extraction and language modeling. By further fine-tuning on a variety of zero/low-resource language generation tasks, including neural machine translation, text summarization and conversational response generation (3 tasks and totally 8 datasets), MASS achieves significant improvements over the baselines without pre-training or with other pre-training methods. Specially, we achieve the state-of-the-art accuracy (37.5 in terms of BLEU score) on the unsupervised English-French translation, even beating the early attention-based supervised model.

cs.CL↗