arXiv subjects

Molei Tao

Publications and source records attributed to Molei Tao.

At least 91 records · Page 5Linked to original sources

arXiv2022

Sqrt(d) Dimension Dependence of Langevin Monte Carlo

This article considers the popular MCMC method of unadjusted Langevin Monte Carlo (LMC) and provides a non-asymptotic analysis of its sampling error in 2-Wasserstein distance. The proof is based on a refinement of mean-square analysis in Li et al. (2019), and this refined framework automates the analysis of a large class of sampling algorithms based on discretizations of contractive SDEs. Using this framework, we establish an $\tilde{O}(\sqrt{d}/ε)$ mixing time bound for LMC, without warm start, under the common log-smooth and log-strongly-convex conditions, plus a growth condition on the 3rd-order derivative of the potential of target measures. This bound improves the best previously known $\tilde{O}(d/ε)$ result and is optimal (in terms of order) in both dimension $d$ and accuracy tolerance $ε$ for target measures satisfying the aforementioned assumptions. Our theoretical analysis is further validated by numerical experiments.

cs.LG↗

arXiv2021

The Mirror Langevin Algorithm Converges with Vanishing Bias

The technique of modifying the geometry of a problem from Euclidean to Hessian metric has proved to be quite effective in optimization, and has been the subject of study for sampling. The Mirror Langevin Diffusion (MLD) is a sampling analogue of mirror flow in continuous time, and it has nice convergence properties under log-Sobolev or Poincare inequalities relative to the Hessian metric, as shown by Chewi et al. (2020). In discrete time, a simple discretization of MLD is the Mirror Langevin Algorithm (MLA) studied by Zhang et al. (2020), who showed a biased convergence bound with a non-vanishing bias term (does not go to zero as step size goes to zero). This raised the question of whether we need a better analysis or a better discretization to achieve a vanishing bias. Here we study the basic Mirror Langevin Algorithm and show it indeed has a vanishing bias. We apply mean-square analysis based on Li et al. (2019) and Li et al. (2021) to show the mixing time bound for MLA under the modified self-concordance condition introduced by Zhang et al. (2020).

cs.DS↗

arXiv2021

Parametric resonance for enhancing the rate of metastable transition

This work is devoted to quantifying how periodic perturbation can change the rate of metastable transition in stochastic mechanical systems with weak noises. A closed-form explicit expression for approximating the rate change is provided, and the corresponding transition mechanism can also be approximated. Unlike the majority of existing relevant works, these results apply to kinetic Langevin equations with high-dimensional potentials and nonlinear perturbations. They are obtained based on a higher-order Hamiltonian formalism and perturbation analysis for the Freidlin-Wentzell action functional. This tool allowed us to show that parametric excitation at a resonant frequency can significantly enhance the rate of metastable transitions. Numerical experiments for both low-dimensional toy models and a molecular cluster are also provided. For the latter, we show that vibrating a material appropriately can help heal its defect, and our theory provides the appropriate vibration.

math.DS↗

arXiv2021

Improving Sampling Accuracy of Stochastic Gradient MCMC Methods via Non-uniform Subsampling of Gradients

Many Markov Chain Monte Carlo (MCMC) methods leverage gradient information of the potential function of target distribution to explore sample space efficiently. However, computing gradients can often be computationally expensive for large scale applications, such as those in contemporary machine learning. Stochastic Gradient (SG-)MCMC methods approximate gradients by stochastic ones, commonly via uniformly subsampled data points, and achieve improved computational efficiency, however at the price of introducing sampling error. We propose a non-uniform subsampling scheme to improve the sampling accuracy. The proposed exponentially weighted stochastic gradient (EWSG) is designed so that a non-uniform-SG-MCMC method mimics the statistical behavior of a batch-gradient-MCMC method, and hence the inaccuracy due to SG approximation is reduced. EWSG differs from classical variance reduction (VR) techniques as it focuses on the entire distribution instead of just the variance; nevertheless, its reduced local variance is also proved. EWSG can also be viewed as an extension of the importance sampling idea, successful for stochastic-gradient-based optimizations, to sampling tasks. In our practical implementation of EWSG, the non-uniform subsampling is performed efficiently via a Metropolis-Hastings chain on the data index, which is coupled to the MCMC algorithm. Numerical experiments are provided, not only to demonstrate EWSG's effectiveness, but also to guide hyperparameter choices, and validate our \emph{non-asymptotic global error bound} despite of approximations in the implementation. Notably, while statistical accuracy is improved, convergence speed can be comparable to the uniform version, which renders EWSG a practical alternative to VR (but EWSG and VR can be combined too).

cs.LG↗

arXiv2021

GRIT: a package for structure-preserving simulations of gravitationally interacting rigid-bodies

Spin-orbit coupling of planetary systems plays an important role in the dynamics and habitability of planets. However, symplectic integrators that can accurately simulate not only how orbit affects spin but also how spin affects orbit have not been constructed for general systems. Thus, we develop symplectic Lie-group integrators to simulate systems consisting gravitationally interacting rigid bodies. A user friendly package (GRIT \url{https://github.com/GRIT-RBSim/GRIT}) is provided and external forcings such as tidal interactions are also included. As a demonstration, this package is applied to Trappist-I. It shows that the differences in transit timing variations due to spin-orbit coupling could reach a few min in ten year measurements, and strong planetary perturbations can push Trappist-I f, g and h out of the synchronized states.

astro-ph.EP↗

arXiv2021

Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Maps

We consider the learning and prediction of nonlinear time series generated by a latent symplectic map. A special case is (not necessarily separable) Hamiltonian systems, whose solution flows give such symplectic maps. For this special case, both generic approaches based on learning the vector field of the latent ODE and specialized approaches based on learning the Hamiltonian that generates the vector field exist. Our method, however, is different as it does not rely on the vector field nor assume its existence; instead, it directly learns the symplectic evolution map in discrete time. Moreover, we do so by representing the symplectic map via a generating function, which we approximate by a neural network (hence the name GFNN). This way, our approximation of the evolution map is always \emph{exactly} symplectic. This additional geometric structure allows the local prediction error at each step to accumulate in a controlled fashion, and we will prove, under reasonable assumptions, that the global prediction error grows at most \emph{linearly} with long prediction time, which significantly improves an otherwise exponential growth. In addition, as a map-based and thus purely data-driven method, GFNN avoids two additional sources of inaccuracies common in vector-field based approaches, namely the error in approximating the vector field by finite difference of the data, and the error in numerical integration of the vector field for making predictions. Numerical experiments further demonstrate our claims.

cs.LG↗

arXiv2021

Variational Symplectic Accelerated Optimization on Lie Groups

There has been significant interest in generalizations of the Nesterov accelerated gradient descent algorithm due to its improved performance guarantee compared to the standard gradient descent algorithm, and its applicability to large scale optimization problems arising in deep learning. A particularly fruitful approach is based on numerical discretizations of differential equations that describe the continuous time limit of the Nesterov algorithm, and a generalization involving time-dependent Bregman Lagrangian and Hamiltonian dynamics that converges at an arbitrarily fast rate to the minimum. We develop a Lie group variational discretization based on an extended path space formulation of the Bregman Lagrangian on Lie groups, and analyze its computational properties with two examples in attitude determination and vision-based localization.

math.OC↗

arXiv2021

Accurate and Efficient Simulations of Hamiltonian Mechanical Systems with Discontinuous Potentials

This article considers Hamiltonian mechanical systems with potential functions admitting jump discontinuities. The focus is on accurate and efficient numerical approximations of their solutions, which will be defined via the laws of reflection and refraction. Despite of the success of symplectic integrators for smooth mechanical systems, their construction for the discontinuous ones is nontrivial, and numerical convergence order can be impaired too. Several rather-usable numerical methods are proposed, including: a first-order symplectic integrator for general problems, a third-order symplectic integrator for problems with only one linear interface, arbitrarily high-order reversible integrators for general problems (no longer symplectic), and an adaptive time-stepping version of the previous high-order method. Interestingly, whether symplecticity leads to favorable long time performance is no longer clear due to discontinuity, as traditional Hamiltonian backward error analysis does not apply any more. Therefore, at this stage, our recommended default method is the last one. Various numerical evidence, on the order of convergence, long time performance, momentum map conservation, and consistency with the computationally-expensive penalty method, are supplied. A complex problem, namely the Sauteed Mushroom, is also proposed and numerically investigated, for which multiple bifurcations between trapped and ergodic dynamics are observed.

math.NA↗

arXiv2020

Why Do Deep Residual Networks Generalize Better than Deep Feedforward Networks? -- A Neural Tangent Kernel Perspective

Deep residual networks (ResNets) have demonstrated better generalization performance than deep feedforward networks (FFNets). However, the theory behind such a phenomenon is still largely unknown. This paper studies this fundamental problem in deep learning from a so-called "neural tangent kernel" perspective. Specifically, we first show that under proper conditions, as the width goes to infinity, training deep ResNets can be viewed as learning reproducing kernel functions with some kernel function. We then compare the kernel of deep ResNets with that of deep FFNets and discover that the class of functions induced by the kernel of FFNets is asymptotically not learnable, as the depth goes to infinity. In contrast, the class of functions induced by the kernel of ResNets does not exhibit such degeneracy. Our discovery partially justifies the advantages of deep ResNets over deep FFNets in generalization abilities. Numerical results are provided to support our claim.

cs.LG↗

arXiv2020

Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Function

This article suggests that deterministic Gradient Descent, which does not use any stochastic gradient approximation, can still exhibit stochastic behaviors. In particular, it shows that if the objective function exhibit multiscale behaviors, then in a large learning rate regime which only resolves the macroscopic but not the microscopic details of the objective, the deterministic GD dynamics can become chaotic and convergent not to a local minimizer but to a statistical distribution. A sufficient condition is also established for approximating this long-time statistical limit by a rescaled Gibbs distribution. Both theoretical and numerical demonstrations are provided, and the theoretical part relies on the construction of a stochastic map that uses bounded noise (as opposed to discretized diffusions).

cs.LG↗

arXiv2020

Variational Optimization on Lie Groups, with Examples of Leading (Generalized) Eigenvalue Problems

The article considers smooth optimization of functions on Lie groups. By generalizing NAG variational principle in vector space (Wibisono et al., 2016) to Lie groups, continuous Lie-NAG dynamics which are guaranteed to converge to local optimum are obtained. They correspond to momentum versions of gradient flow on Lie groups. A particular case of $\mathsf{SO}(n)$ is then studied in details, with objective functions corresponding to leading Generalized EigenValue problems: the Lie-NAG dynamics are first made explicit in coordinates, and then discretized in structure preserving fashions, resulting in optimization algorithms with faithful energy behavior (due to conformal symplecticity) and exactly remaining on the Lie group. Stochastic gradient versions are also investigated. Numerical experiments on both synthetic data and practical problem (LDA for MNIST) demonstrate the effectiveness of the proposed methods as optimization algorithms ($not$ as a classification method).

cs.LG↗

arXiv2019

Space-Time Phononic Crystals with Anomalous Topological Edge States

It is well known that an interface created by two topologically distinct structures could host nontrivial edge states that are immune to defects. In this letter, we introduce a one-dimensional space-time phononic crystal and study the associated anomalous topological edge states. A space-decoupled time modulation is assumed. While preserving the key topological feature of the system, such a modulation also duplicates the edge state mode across the spectrum, both inside and outside the band gap. It is shown that, in contrast to conventional topological edge states which are excited by frequencies in the Bragg regime, the time-modulation-induced frequency conversion can be leveraged to access topological edge states at a deep subwavelength scale where the entire phononic crystal size is merely 1/5.1 of the wavelength. This remarkable feature could open a new route for designing miniature devices that are based on topological physics.

physics.class-ph↗

arXiv2019

Dynamic mode decomposition for multiscale nonlinear physics

We present a data-driven method for separating complex, multiscale systems into their constituent time-scale components using a recursive implementation of dynamic mode decomposition (DMD). Local linear models are built from windowed subsets of the data, and dominant time scales are discovered using spectral clustering on their eigenvalues. This approach produces time series data for each identified component, which sum to a faithful reconstruction of the input signal. It differs from most other methods in the field of multiresolution analysis (MRA) in that it 1) accounts for spatial and temporal coherencies simultaneously, making it more robust to scale overlap between components, and 2) yields a closed-form expression for local dynamics at each scale, which can be used for short-term prediction of any or all components. Our technique is an extension of multi-resolution dynamic mode decomposition (mrDMD), generalized to treat a broader variety of multiscale systems and more faithfully reconstruct their isolated components. In this paper we present an overview of our algorithm and its results on two example physical systems, and briefly discuss some advantages and potential forecasting applications for the technique.

eess.SY↗

arXiv2018

Simply improved averaging for coupled oscillators and weakly nonlinear waves

The long time effect of nonlinear perturbation to oscillatory linear systems can be characterized by the averaging method, and we consider first-order averaging for its simplest applicability to high-dimensional problems. Instead of the classical approach, in which one uses the pullback of linear flow to isolate slow variables and then approximate the effective dynamics by averaging, we propose an alternative coordinate transform that better approximates the mean of oscillations. This leads to a simple improvement of the averaged system, which will be shown both theoretically and numerically to provide a more accurate approximation. Three examples are then provided: in the first, a new device for wireless energy transfer modeled by two coupled oscillators was analyzed, and the results provide design guidance and performance quantification for the device; the second is a classical coupled oscillator problem (Fermi-Pasta-Ulam), for which we numerically observed improved accuracy beyond the theoretically justified timescale; the third is a nonlinearly perturbed first-order wave equation, which demonstrates the efficacy of improved averaging in an infinite dimensional setting.

math.CA↗

arXiv2018

Metastable transitions in inertial Langevin systems: what can be different from the overdamped case?

Metastable transitions in Langevin dynamics can exhibit rich behaviors that are markedly different from its overdamped limit. In addition to local alterations of the transition path geometry, more fundamental global changes may exist. For instance, when the dissipation is weak, heteroclinic connections that exist in the overdamped limit do not necessarily have a counterpart in the Langevin system, potentially leading to different transition rates. Furthermore, when the friction coefficient depends on the velocity, the overdamped limit no longer exists, but it is still possible to efficiently find instantons. The approach we employed for these discoveries was based on (i) a simple rewriting of the Freidlin-Wentzell action in terms of time-reversed dynamics, and (ii) an adaptation of the string method, which was originally designed for gradient systems, to this specific non-gradient system.

physics.comp-ph↗

arXiv2017

Hyperbolic periodic orbits in nongradient systems and small-noise-induced metastable transitions

Small noise can induce rare transitions between metastable states, which can be characterized by Maximum Likelihood Paths (MLPs). Nongradient systems contrast gradient systems in that MLP does not have to cross the separatrix at a saddle point, but instead possibly at a point on a hyperbolic periodic orbit. A numerical approach for identifying such unstable periodic orbits is proposed based on String method. In a special class of nongradient systems (`orthogonal-type'), there are provably local MLPs that cross such saddle point or hyperbolic periodic orbit, and the separatrix crossing location determines the associated local maximum of transition rate. In general cases, however, the separatrix crossing may not determine a unique local maximum of the rate, as we numerically observed a counter-example in a sheared 2D-space Allen-Cahn SPDE. It is a reasonable conjecture that there are always local MLPs associated with each attractor on the separatrix, such as saddle point or hyperbolic periodic orbit; our numerical experiments did not disprove so.

math.DS↗

arXiv2016

Explicit symplectic approximation of nonseparable Hamiltonians: algorithm and long time performance

Explicit symplectic integrators have been important tools for accurate and efficient approximations of mechanical systems with separable Hamiltonians. For the first time, the article proposes for arbitrary Hamiltonians similar integrators, which are explicit, of any even order, symplectic in an extended phase space, and with pleasant long time properties. They are based on a mechanical restraint that binds two copies of phase space together. Using backward error analysis, KAM theory, and additional multiscale analysis, an error bound of $\mathcal{O}(Tδ^l ω)$ is established for integrable systems, where $T$, $δ$, $l$ and $ω$ are respectively the (long) simulation time, step size, integrator order, and some binding constant. For non-integrable systems with positive Lyapunov exponents, such an error bound is generally impossible, but satisfactory statistical behaviors were observed in a numerical experiment with a nonlinear Schrödinger equation.

math.NA↗

arXiv2016

Uncovering Circumbinary Planetary Architectural Properties from Selection Biases

The new discoveries of circumbinary planetary systems shed light on the understanding of planetary system formation. Learning the architectural properties of these systems is essential for constraining the different formation mechanisms. We first revisit the stability limit of circumbinary planets. Next, we focus on eclipsing stellar binaries and obtain an analytical expression for the transit probability in a realistic setting, where finite observation period and planetary orbital precession are included. Then, understanding of the architectural properties of the currently observed transiting systems is refined, based on Bayesian analysis and a series of hypothesis tests. We find 1) it is not a selection bias that the innermost planets reside near the stability limit for eight of the nine observed systems, and this is consistent with a log uniform distribution of the planetary semi-major axis; 2) it is not a selection bias that the planetary and stellar orbits are nearly coplanar ($\lesssim 3^\circ$), and this together with previous studies may imply an occurrence rate of circumbinary planets similar to that of single star systems; 3) the dominance of observed circumbinary systems with only one transiting planet may be caused by selection effects; 4) formation mechanisms involving Lidov-Kozai oscillations, which may produce misalignment and large separation between planet and stellar binaries, are consistent with the lack of transiting circumbinary planets around short-period stellar binaries, in agreement with previous studies. As a consequence of 4), eclipse timing variations may better suit the detection of planets in such configurations.

astro-ph.EP↗