arXiv subjects

J. Nathan Kutz

Publications and source records attributed to J. Nathan Kutz.

At least 127 records · Page 7Linked to original sources

arXiv2022

Dynamic Mode Decomposition for Aero-Optic Wavefront Characterization

Aero-optical beam control relies on the development of low-latency forecasting techniques to quickly predict wavefronts aberrated by the Turbulent Boundary Layer (TBL) around an airborne optical system, and its study applies to a multi-domain need from astronomy to microscopy for high-fidelity laser propagation. We leverage the forecasting capabilities of the Dynamic Mode Decomposition (DMD) -- an equation-free, data-driven method for identifying coherent flow structures and their associated spatiotemporal dynamics -- in order to estimate future state wavefront phase aberrations to feed into an adaptive optic (AO) control loop. We specifically leverage the optimized DMD (opt-DMD) algorithm on a subset of the Airborne Aero-Optics Laboratory Transonic (AAOL-T) experimental dataset, characterizing aberrated wavefront dynamics for 23 beam propagation directions via the spatiotemporal decomposition underlying DMD. Critically, we show that opt-DMD produces an optimally de-biased eigenvalue spectrum with imaginary eigenvalues, allowing for arbitrarily long forecasting to produce a robust future-state prediction, while exact DMD loses structural information due to modal decay rates.

physics.flu-dyn↗

arXiv2022

Principal component trajectories for modeling spectrally-continuous dynamics as forced linear systems

Delay embeddings of time series data have emerged as a promising coordinate basis for data-driven estimation of the Koopman operator, which seeks a linear representation for observed nonlinear dynamics. Recent work has demonstrated the efficacy of Dynamic Mode Decomposition (DMD) for obtaining finite-dimensional Koopman approximations in delay coordinates. In this paper we demonstrate how nonlinear dynamics with sparse Fourier spectra can be (i) represented by a superposition of principal component trajectories (PCT) and (ii) modeled by DMD in this coordinate space. For continuous or mixed (discrete and continuous) spectra, DMD can be augmented with an external forcing term. We present a method for learning linear control models in delay coordinates while simultaneously discovering the corresponding exogeneous forcing signal in a fully unsupervised manner. This extends the existing DMD with control (DMDc) algorithm to cases where a control signal is not known a priori. We provide examples to validate the learned forcing against a known ground truth and illustrate their statistical similarity. Finally we offer a demonstration of this method applied to real-world power grid load data to show its utility for diagnostics and interpretation on systems in which somewhat periodic behavior is strongly forced by unknown and unmeasurable environmental variables.

physics.comp-ph↗

arXiv2022

Pruning deep neural networks generates a sparse, bio-inspired nonlinear controller for insect flight

Insect flight is a strongly nonlinear and actuated dynamical system. As such, strategies for understanding its control have typically relied on either model-based methods or linearizations thereof. Here we develop a framework that combines model predictive control on an established flight dynamics model and deep neural networks (DNN) to create an efficient method for solving the inverse problem of flight control. We turn to natural systems for inspiration since they inherently demonstrate network pruning with the consequence of yielding more efficient networks for a specific set of tasks. This bio-inspired approach allows us to leverage network pruning to optimally sparsify a DNN architecture in order to perform flight tasks with as few neural connections as possible, however, there are limits to sparsification. Specifically, as the number of connections falls below a critical threshold, flight performance drops considerably. We develop sparsification paradigms and explore their limits for control tasks. Monte Carlo simulations also quantify the statistical distribution of network weights during pruning given initial random weights of the DNNs. We demonstrate that on average, the network can be pruned to retain approximately 7% of the original network weights, with statistical distributions quantified at each layer of the network. Overall, this work shows that sparsely connected DNNs are capable of predicting the forces required to follow flight trajectories. Additionally, sparsification has sharp performance limits.

q-bio.QM↗

arXiv2021

A toolkit for data-driven discovery of governing equations in high-noise regimes

We consider the data-driven discovery of governing equations from time-series data in the limit of high noise. The algorithms developed describe an extensive toolkit of methods for circumventing the deleterious effects of noise in the context of the sparse identification of nonlinear dynamics (SINDy) framework. We offer two primary contributions, both focused on noisy data acquired from a system x' = f(x). First, we propose, for use in high-noise settings, an extensive toolkit of critically enabling extensions for the SINDy regression method, to progressively cull functionals from an over-complete library and yield a set of sparse equations that regress to the derivate x'. These innovations can extract sparse governing equations and coefficients from high-noise time-series data (e.g. 300% added noise). For example, it discovers the correct sparse libraries in the Lorenz system, with median coefficient estimate errors equal to 1% - 3% (for 50% noise), 6% - 8% (for 100% noise); and 23% - 25% (for 300% noise). The enabling modules in the toolkit are combined into a single method, but the individual modules can be tactically applied in other equation discovery methods (SINDy or not) to improve results on high-noise data. Second, we propose a technique, applicable to any model discovery method based on x' = f(x), to assess the accuracy of a discovered model in the context of non-unique solutions due to noisy data. Currently, this non-uniqueness can obscure a discovered model's accuracy and thus a discovery method's effectiveness. We describe a technique that uses linear dependencies among functionals to transform a discovered model into an equivalent form that is closest to the true model, enabling more accurate assessment of a discovered model's accuracy.

cs.LG↗

arXiv2021

Physics-informed dynamic mode decomposition (piDMD)

In this work, we demonstrate how physical principles -- such as symmetries, invariances, and conservation laws -- can be integrated into the dynamic mode decomposition (DMD). DMD is a widely-used data analysis technique that extracts low-rank modal structures and dynamics from high-dimensional measurements. However, DMD frequently produces models that are sensitive to noise, fail to generalize outside the training data, and violate basic physical laws. Our physics-informed DMD (piDMD) optimization, which may be formulated as a Procrustes problem, restricts the family of admissible models to a matrix manifold that respects the physical structure of the system. We focus on five fundamental physical principles -- conservation, self-adjointness, localization, causality, and shift-invariance -- and derive several closed-form solutions and efficient algorithms for the corresponding piDMD optimizations. With fewer degrees of freedom, piDMD models are less prone to overfitting, require less training data, and are often less computationally expensive to build than standard DMD models. We demonstrate piDMD on a range of challenging problems in the physical sciences, including energy-preserving fluid flow, travelling-wave systems, the Schrödinger equation, solute advection-diffusion, a system with causal dynamics, and three-dimensional transitional channel flow. In each case, piDMD significantly outperforms standard DMD in metrics such as spectral identification, state prediction, and estimation of optimal forcings and responses.

math.DS↗

arXiv2021

Ensemble-SINDy: Robust sparse model discovery in the low-data, high-noise limit, with active learning and control

Sparse model identification enables the discovery of nonlinear dynamical systems purely from data; however, this approach is sensitive to noise, especially in the low-data limit. In this work, we leverage the statistical approach of bootstrap aggregating (bagging) to robustify the sparse identification of nonlinear dynamics (SINDy) algorithm. First, an ensemble of SINDy models is identified from subsets of limited and noisy data. The aggregate model statistics are then used to produce inclusion probabilities of the candidate functions, which enables uncertainty quantification and probabilistic forecasts. We apply this ensemble-SINDy (E-SINDy) algorithm to several synthetic and real-world data sets and demonstrate substantial improvements to the accuracy and robustness of model discovery from extremely noisy and limited data. For example, E-SINDy uncovers partial differential equations models from data with more than twice as much measurement noise as has been previously reported. Similarly, E-SINDy learns the Lotka Volterra dynamics from remarkably limited data of yearly lynx and hare pelts collected from 1900-1920. E-SINDy is computationally efficient, with similar scaling as standard SINDy. Finally, we show that ensemble statistics from E-SINDy can be exploited for active learning and improved model predictive control.

math.NA↗

arXiv2021

FC2T2: The Fast Continuous Convolutional Taylor Transform with Applications in Vision and Graphics

Series expansions have been a cornerstone of applied mathematics and engineering for centuries. In this paper, we revisit the Taylor series expansion from a modern Machine Learning perspective. Specifically, we introduce the Fast Continuous Convolutional Taylor Transform (FC2T2), a variant of the Fast Multipole Method (FMM), that allows for the efficient approximation of low dimensional convolutional operators in continuous space. We build upon the FMM which is an approximate algorithm that reduces the computational complexity of N-body problems from O(NM) to O(N+M) and finds application in e.g. particle simulations. As an intermediary step, the FMM produces a series expansion for every cell on a grid and we introduce algorithms that act directly upon this representation. These algorithms analytically but approximately compute the quantities required for the forward and backward pass of the backpropagation algorithm and can therefore be employed as (implicit) layers in Neural Networks. Specifically, we introduce a root-implicit layer that outputs surface normals and object distances as well as an integral-implicit layer that outputs a rendering of a radiance field given a 3D pose. In the context of Machine Learning, $N$ and $M$ can be understood as the number of model parameters and model evaluations respectively which entails that, for applications that require repeated function evaluations which are prevalent in Computer Vision and Graphics, unlike regular Neural Networks, the techniques introduce in this paper scale gracefully with parameters. For some applications, this results in a 200x reduction in FLOPs compared to state-of-the-art approaches at a reasonable or non-existent loss in accuracy.

cs.LG↗

arXiv2021

Modern Koopman Theory for Dynamical Systems

The field of dynamical systems is being transformed by the mathematical tools and algorithms emerging from modern computing and data science. First-principles derivations and asymptotic reductions are giving way to data-driven approaches that formulate models in operator theoretic or probabilistic frameworks. Koopman spectral theory has emerged as a dominant perspective over the past decade, in which nonlinear dynamics are represented in terms of an infinite-dimensional linear operator acting on the space of all possible measurement functions of the system. This linear representation of nonlinear dynamics has tremendous potential to enable the prediction, estimation, and control of nonlinear systems with standard textbook methods developed for linear systems. However, obtaining finite-dimensional coordinate systems and embeddings in which the dynamics appear approximately linear remains a central open challenge. The success of Koopman analysis is due primarily to three key factors: 1) there exists rigorous theory connecting it to classical geometric approaches for dynamical systems, 2) the approach is formulated in terms of measurements, making it ideal for leveraging big-data and machine learning techniques, and 3) simple, yet powerful numerical algorithms, such as the dynamic mode decomposition (DMD), have been developed and extended to reduce Koopman theory to practice in real-world applications. In this review, we provide an overview of modern Koopman operator theory, describing recent theoretical and algorithmic developments and highlighting these methods with a diverse range of applications. We also discuss key advances and challenges in the rapidly growing field of machine learning that are likely to drive future developments and significantly transform the theoretical landscape of dynamical systems.

math.DS↗

arXiv2021

Data-driven stochastic modeling of coarse-grained dynamics with finite-size effects using Langevin regression

Obtaining coarse-grained models that accurately incorporate finite-size effects is an important open challenge in the study of complex, multi-scale systems. We apply Langevin regression, a recently developed method for finding stochastic differential equation (SDE) descriptions of realistically-sampled time series data, to understand finite-size effects in the Kuramoto model of coupled oscillators. We find that across the entire bifurcation diagram, the dynamics of the Kuramoto order parameter are statistically consistent with an SDE whose drift term has the form predicted by the Ott-Antonsen ansatz in the $N\to \infty$ limit. We find that the diffusion term is nearly independent of the bifurcation parameter, and has a magnitude decaying as $N^{-1/2}$, consistent with the central limit theorem. This shows that the diverging fluctuations of the order parameter near the critical point are driven by a bifurcation in the underlying drift term, rather than increased stochastic forcing.

nlin.AO↗

arXiv2021

SINDy with Control: A Tutorial

Many dynamical systems of interest are nonlinear, with examples in turbulence, epidemiology, neuroscience, and finance, making them difficult to control using linear approaches. Model predictive control (MPC) is a powerful model-based optimization technique that enables the control of such nonlinear systems with constraints. However, modern systems often lack computationally tractable models, motivating the use of system identification techniques to learn accurate and efficient models for real-time control. In this tutorial article, we review emerging data-driven methods for model discovery and how they are used for nonlinear MPC. In particular, we focus on the sparse identification of nonlinear dynamics (SINDy) algorithm and show how it may be used with MPC on an infectious disease control example. We compare the performance against MPC based on a linear dynamic mode decomposition (DMD) model. Code is provided to run the tutorial examples and may be modified to extend this data-driven control framework to arbitrary nonlinear systems.

math.OC↗

arXiv2021

Robust Trimmed k-means

Clustering is a fundamental tool in unsupervised learning, used to group objects by distinguishing between similar and dissimilar features of a given data set. One of the most common clustering algorithms is k-means. Unfortunately, when dealing with real-world data many traditional clustering algorithms are compromised by lack of clear separation between groups, noisy observations, and/or outlying data points. Thus, robust statistical algorithms are required for successful data analytics. Current methods that robustify k-means clustering are specialized for either single or multi-membership data, but do not perform competitively in both cases. We propose an extension of the k-means algorithm, which we call Robust Trimmed k-means (RTKM) that simultaneously identifies outliers and clusters points and can be applied to either single- or multi-membership data. We test RTKM on various real-world datasets and show that RTKM performs competitively with other methods on single membership data with outliers and multi-membership data without outliers. We also show that RTKM leverages its relative advantages to outperform other methods on multi-membership data containing outliers.

stat.ML↗

arXiv2021

Bagging, optimized dynamic mode decomposition (BOP-DMD) for robust, stable forecasting with spatial and temporal uncertainty-quantification

Dynamic mode decomposition (DMD) provides a regression framework for adaptively learning a best-fit linear dynamics model over snapshots of temporal, or spatio-temporal, data. A diversity of regression techniques have been developed for producing the linear model approximation whose solutions are exponentials in time. For spatio-temporal data, DMD provides low-rank and interpretable models in the form of dominant modal structures along with their exponential/oscillatory behavior in time. The majority of DMD algorithms, however, are prone to bias errors from noisy measurements of the dynamics, leading to poor model fits and unstable forecasting capabilities. The optimized DMD algorithm minimizes the model bias with a variable projection optimization, thus leading to stabilized forecasting capabilities. Here, the optimized DMD algorithm is improved by using statistical bagging methods whereby a single set of snapshots is used to produce an ensemble of optimized DMD models. The outputs of these models are averaged to produce a bagging, optimized dynamic mode decomposition (BOP-DMD). BOP-DMD not only improves performance, it also robustifies the model and provides both spatial and temporal uncertainty quantification (UQ). Thus unlike currently available DMD algorithms, BOP-DMD provides a stable and robust model for probabilistic, or Bayesian forecasting with comprehensive UQ metrics.

cs.LG↗

arXiv2021

Deep Learning of Conjugate Mappings

Despite many of the most common chaotic dynamical systems being continuous in time, it is through discrete time mappings that much of the understanding of chaos is formed. Henri Poincaré first made this connection by tracking consecutive iterations of the continuous flow with a lower-dimensional, transverse subspace. The mapping that iterates the dynamics through consecutive intersections of the flow with the subspace is now referred to as a Poincaré map, and it is the primary method available for interpreting and classifying chaotic dynamics. Unfortunately, in all but the simplest systems, an explicit form for such a mapping remains outstanding. This work proposes a method for obtaining explicit Poincaré mappings by using deep learning to construct an invertible coordinate transformation into a conjugate representation where the dynamics are governed by a relatively simple chaotic mapping. The invertible change of variable is based on an autoencoder, which allows for dimensionality reduction, and has the advantage of classifying chaotic systems using the equivalence relation of topological conjugacies. Indeed, the enforcement of topological conjugacies is the critical neural network regularization for learning the coordinate and dynamics pairing. We provide expository applications of the method to low-dimensional systems such as the Rössler and Lorenz systems, while also demonstrating the utility of the method on infinite-dimensional systems, such as the Kuramoto--Sivashinsky equation.

math.DS↗

arXiv2021

Stochastically forced ensemble dynamic mode decomposition for forecasting and analysis of near-periodic systems

Time series forecasting remains a central challenge problem in almost all scientific disciplines. We introduce a novel load forecasting method in which observed dynamics are modeled as a forced linear system using Dynamic Mode Decomposition (DMD) in time delay coordinates. Central to this approach is the insight that grid load, like many observables on complex real-world systems, has an "almost-periodic" character, i.e., a continuous Fourier spectrum punctuated by dominant peaks, which capture regular (e.g., daily or weekly) recurrences in the dynamics. The forecasting method presented takes advantage of this property by (i) regressing to a deterministic linear model whose eigenspectrum maps onto those peaks, and (ii) simultaneously learning a stochastic Gaussian process regression (GPR) process to actuate this system. Our forecasting algorithm is compared against state-of-the-art forecasting techniques not using additional explanatory variables and is shown to produce superior performance. Moreover, its use of linear intrinsic dynamics offers a number of desirable properties in terms of interpretability and parsimony. Results are presented for a test case using load data from an electrical grid. Load forecasting is an essential challenge in power systems engineering, with major implications for real-time control, pricing, maintenance, and security decisions.

physics.soc-ph↗

arXiv2021

Sparsifying Priors for Bayesian Uncertainty Quantification in Model Discovery

We propose a probabilistic model discovery method for identifying ordinary differential equations (ODEs) governing the dynamics of observed multivariate data. Our method is based on the sparse identification of nonlinear dynamics (SINDy) framework, in which target ODE models are expressed as a sparse linear combinations of pre-specified candidate functions. Promoting parsimony through sparsity in SINDy leads to interpretable models that generalize to unknown data. Instead of targeting point estimates of the SINDy (linear combination) coefficients, in this work we estimate these coefficients via sparse Bayesian inference. The resulting method, uncertainty quantification SINDy (UQ-SINDy), quantifies not only the uncertainty in the values of the SINDy coefficients due to observation errors and limited data, but also the probability of inclusion of each candidate function in the linear combination. UQ-SINDy promotes robustness against observation noise and limited data, interpretability (in terms of model selection and inclusion probabilities), and generalization capacity for out-of-sample forecast. Sparse inference for UQ-SINDy employs Markov Chain Monte Carlo, and we explore two sparsifying priors: the spike-and-slab prior, and the regularized horseshoe prior. We apply UQ-SINDy to synthetic nonlinear data sets from a Lotka-Volterra model and a nonlinear oscillator, and to a real-world data set of lynx and hare populations. We find that UQ-SINDy is able to discover accurate and meaningful models even in the presence of noise and limited data samples.

math.DS↗

arXiv2021

Data-driven Modeling of Two-Dimensional Detonation Wave Fronts

Historical experimental testing of high-altitude nuclear explosions (HANEs) are known to cause severe and detrimental effects to radio frequency signals and communications infrastructure. In order to study and predict the impact of HANEs, tractable computational approaches are required to model the complex physical processes involved in the detonation wave physics. Modern reduced-order models (ROMs) can enable long-time and many-parameter simulations with minimal computational cost. However, translational and scale invariances inherent to this type of wave propagation problem are known to limit traditional ROM approaches. Specifically, dimensionality reduction methods are typically ineffective in producing low-rank models when invariances are present in the data. In this work, an unsupervised machine learning method is used to discover coordinate systems that make such invariances amenable to traditional dimensionality reduction methods. The method, which has previously been demonstrated on one-dimensional translations, is extended to higher dimensions and additional invariances. A surrogate HANE system, i.e. a HANE-ROM, with one detonation wave is captured well at extremely low-rank. Two detonation-waves are also considered with various amounts of interaction between the waves, with improvements to low-rank models for multiple wave quantities with limited interaction.

physics.flu-dyn↗

arXiv2021

Deep Probabilistic Koopman: Long-term time-series forecasting under periodic uncertainties

Probabilistic forecasting of complex phenomena is paramount to various scientific disciplines and applications. Despite the generality and importance of the problem, general mathematical techniques that allow for stable long-term forecasts with calibrated uncertainty measures are lacking. For most time series models, the difficulty of obtaining accurate probabilistic future time step predictions increases with the prediction horizon. In this paper, we introduce a surprisingly simple approach that characterizes time-varying distributions and enables reasonably accurate predictions thousands of timesteps into the future. This technique, which we call Deep Probabilistic Koopman (DPK), is based on recent advances in linear Koopman operator theory, and does not require time stepping for future time predictions. Koopman models also tend to have a small parameter footprint (often less than 10,000 parameters). We demonstrate the long-term forecasting performance of these models on a diversity of domains, including electricity demand forecasting, atmospheric chemistry, and neuroscience. For electricity demand modeling, our domain-agnostic technique outperforms all of 177 domain-specific competitors in the most recent Global Energy Forecasting Competition.

cs.LG↗

arXiv2021

Learning normal form autoencoders for data-driven discovery of universal,parameter-dependent governing equations

Complex systems manifest a small number of instabilities and bifurcations that are canonical in nature, resulting in universal pattern forming characteristics as a function of some parametric dependence. Such parametric instabilities are mathematically characterized by their universal un-foldings, or normal form dynamics, whereby a parsimonious model can be used to represent the dynamics. Although center manifold theory guarantees the existence of such low-dimensional normal forms, finding them has remained a long standing challenge. In this work, we introduce deep learning autoencoders to discover coordinate transformations that capture the underlying parametric dependence of a dynamical system in terms of its canonical normal form, allowing for a simple representation of the parametric dependence and bifurcation structure. The autoencoder constrains the latent variable to adhere to a given normal form, thus allowing it to learn the appropriate coordinate transformation. We demonstrate the method on a number of example problems, showing that it can capture a diverse set of normal forms associated with Hopf, pitchfork, transcritical and/or saddle node bifurcations. This method shows how normal forms can be leveraged as canonical and universal building blocks in deep learning approaches for model discovery and reduced-order modeling.

cs.LG↗