arXiv subjects

Boumediene Hamzi

Publications and source records attributed to Boumediene Hamzi.

36 records · Page 2Linked to original sources

arXiv2023

Learning Dynamical Systems from Data: A Simple Cross-Validation Perspective, Part V: Sparse Kernel Flows for 132 Chaotic Dynamical Systems

Regressing the vector field of a dynamical system from a finite number of observed states is a natural way to learn surrogate models for such systems. A simple and interpretable way to learn a dynamical system from data is to interpolate its vector-field with a data-adapted kernel which can be learned by using Kernel Flows. The method of Kernel Flows is a trainable machine learning method that learns the optimal parameters of a kernel based on the premise that a kernel is good if there is no significant loss in accuracy if half of the data is used. The objective function could be a short-term prediction or some other objective for other variants of Kernel Flows). However, this method is limited by the choice of the base kernel. In this paper, we introduce the method of \emph{Sparse Kernel Flows } in order to learn the ``best'' kernel by starting from a large dictionary of kernels. It is based on sparsifying a kernel that is a linear combination of elemental kernels. We apply this approach to a library of 132 chaotic systems.

stat.ML↗

arXiv2022

One-Shot Learning of Stochastic Differential Equations with Data Adapted Kernels

We consider the problem of learning Stochastic Differential Equations of the form $dX_t = f(X_t)dt+σ(X_t)dW_t $ from one sample trajectory. This problem is more challenging than learning deterministic dynamical systems because one sample trajectory only provides indirect information on the unknown functions $f$, $σ$, and stochastic process $dW_t$ representing the drift, the diffusion, and the stochastic forcing terms, respectively. We propose a method that combines Computational Graph Completion and data adapted kernels learned via a new variant of cross validation. Our approach can be decomposed as follows: (1) Represent the time-increment map $X_t \rightarrow X_{t+dt}$ as a Computational Graph in which $f$, $σ$ and $dW_t$ appear as unknown functions and random variables. (2) Complete the graph (approximate unknown functions and random variables) via Maximum a Posteriori Estimation (given the data) with Gaussian Process (GP) priors on the unknown functions. (3) Learn the covariance functions (kernels) of the GP priors from data with randomized cross-validation. Numerical experiments illustrate the efficacy, robustness, and scope of our method.

stat.ML↗

arXiv2022

A Note on A Priori Forecasting and Simplicity Bias in Time Series

To what extent can we forecast a time series without fitting to historical data? Can universal patterns of probability help in this task? Deep relations between pattern Kolmogorov complexity and pattern probability have recently been used to make \emph{a priori} probability predictions in a variety of systems in physics, biology and engineering. Here we study \emph{simplicity bias} (SB) -- an exponential upper bound decay in pattern probability with increasing complexity -- in discretised time series extracted from the World Bank Open Data collection. We predict upper bounds on the probability of discretised series patterns, without fitting to trends in the data. Thus we perform a kind of `forecasting without training data', predicting time series shape patterns \emph{a priori}, but not the actual numerical value of the series. Additionally we make predictions about which of two discretised series is more likely with accuracy of $\sim$80\%, much higher than a 50\% baseline rate, just by using the complexity of each series. These results point to a promising perspective on practical time series forecasting and integration with machine learning methods.

physics.data-an↗

arXiv2021

Data-driven geophysical forecasting: Simple, low-cost, and accurate baselines with kernel methods

Modeling geophysical processes as low-dimensional dynamical systems and regressing their vector field from data is a promising approach for learning emulators of such systems. We show that when the kernel of these emulators is also learned from data (using kernel flows, a variant of cross-validation), then the resulting data-driven models are not only faster than equation-based models but are easier to train than neural networks such as the long short-term memory neural network. In addition, they are also more accurate and predictive than the latter. When trained on geophysical observational data, for example, the weekly averaged global sea-surface temperature, considerable gains are also observed by the proposed technique in comparison to classical partial differential equation-based models in terms of forecast computational cost and accuracy. When trained on publicly available re-analysis data for the daily temperature of the North-American continent, we see significant improvements over classical baselines such as climatology and persistence-based forecast techniques. Although our experiments concern specific examples, the proposed approach is general, and our results support the viability of kernel methods (with learned kernels) for interpretable and computationally efficient geophysical forecasting for a large diversity of processes.

physics.ao-ph↗

arXiv2021

Entropy for practical stabilization

For deterministic continuous time nonlinear control systems, epsilon-practical stabilization entropy and practical stabilization entropy are introduced. Here the rate of attraction is specified by a KL-function. Upper and lower bounds for the diverse entropies are proved, with special attention to exponential KL-functions. The relation to feedbacks is discussed, the linear case and several nonlinear examples are analyzed in detail.

math.OC↗

arXiv2020

Kernel-based approximation of the Koopman generator and Schrödinger operator

Many dimensionality and model reduction techniques rely on estimating dominant eigenfunctions of associated dynamical operators from data. Important examples include the Koopman operator and its generator, but also the Schrödinger operator. We propose a kernel-based method for the approximation of differential operators in reproducing kernel Hilbert spaces and show how eigenfunctions can be estimated by solving auxiliary matrix eigenvalue problems. The resulting algorithms are applied to molecular dynamics and quantum chemistry examples. Furthermore, we exploit that, under certain conditions, the Schrödinger operator can be transformed into a Kolmogorov backward operator corresponding to a drift-diffusion process and vice versa. This allows us to apply methods developed for the analysis of high-dimensional stochastic differential equations to quantum mechanical systems.

math.DS↗

arXiv2020

Kernel methods for center manifold approximation and a data-based version of the Center Manifold Theorem

For dynamical systems with a non hyperbolic equilibrium, it is possible to significantly simplify the study of stability by means of the center manifold theory. This theory allows to isolate the complicated asymptotic behavior of the system close to the equilibrium point and to obtain meaningful predictions of its behavior by analyzing a reduced order system on the so-called center manifold. Since the center manifold is usually not known, good approximation methods are important as the center manifold theorem states that the stability properties of the origin of the reduced order system are the same as those of the origin of the full order system. In this work, we establish a data-based version of the center manifold theorem that works by considering an approximation in place of an exact manifold. Also the error between the approximated and the original reduced dynamics are quantified. We then use an apposite data-based kernel method to construct a suitable approximation of the manifold close to the equilibrium, which is compatible with our general error theory. The data are collected by repeated numerical simulation of the full system by means of a high-accuracy solver, which generates sets of discrete trajectories that are then used as a training set. The method is tested on different examples which show promising performance and good accuracy.

math.NA↗

arXiv2020

Learning dynamical systems from data: a simple cross-validation perspective

Regressing the vector field of a dynamical system from a finite number of observed states is a natural way to learn surrogate models for such systems. We present variants of cross-validation (Kernel Flows \cite{Owhadi19} and its variants based on Maximum Mean Discrepancy and Lyapunov exponents) as simple approaches for learning the kernel used in these emulators.

cs.LG↗

arXiv2020

Dimensionality Reduction of Complex Metastable Systems via Kernel Embeddings of Transition Manifolds

We present a novel kernel-based machine learning algorithm for identifying the low-dimensional geometry of the effective dynamics of high-dimensional multiscale stochastic systems. Recently, the authors developed a mathematical framework for the computation of optimal reaction coordinates of such systems that is based on learning a parametrization of a low-dimensional transition manifold in a certain function space. In this article, we enhance this approach by embedding and learning this transition manifold in a reproducing kernel Hilbert space, exploiting the favorable properties of kernel embeddings. Under mild assumptions on the kernel, the manifold structure is shown to be preserved under the embedding, and distortion bounds can be derived. This leads to a more robust and more efficient algorithm compared to previous parametrization approaches.

math.DS↗

arXiv2018

A Note on Local Mode-in-State Participation Factors for Nonlinear Systems

The paper studies an extension to nonlinear systems of a recently proposed approach to the concept of modal participation factors. First, a definition is given for local mode-in-state participation factors for smooth nonlinear autonomous systems. The definition is general, and, unlike in the more traditional approach, the resulting participation measures depend on the assumed uncertainty law governing the system initial condition. The work follows Hashlamoun, Hassouneh and Abed (2009) in taking a mathematical expectation (or set-theoretic average) of a modal contribution measure with respect to an assumed uncertain initial state. As in the linear case, it is found that a symmetry assumption on the distribution of the initial state results in a tractable calculation and an explicit and simple formula for mode-in-state participation factors.

eess.SY↗

arXiv2018

Greedy Kernel Methods for Center Manifold Approximation

For certain dynamical systems it is possible to significantly simplify the study of stability by means of the center manifold theory. This theory allows to isolate the complicated asymptotic behavior of the system close to a non-hyperbolic equilibrium point, and to obtain meaningful predictions of its behavior by analyzing a reduced dimensional problem. Since the manifold is usually not known, approximation methods are of great interest to obtain qualitative estimates. In this work, we use a data-based greedy kernel method to construct a suitable approximation of the manifold close to the equilibrium. The data are collected by repeated numerical simulation of the full system by means of a high-accuracy solver, which generates sets of discrete trajectories that are then used to construct a surrogate model of the manifold. The method is tested on different examples which show promising performance and good accuracy.

math.DS↗

arXiv2018

A Note on Kernel Methods for Multiscale Systems with Critical Transitions

We study the maximum mean discrepancy (MMD) in the context of critical transitions modelled by fast-slow stochastic dynamical systems. We establish a new link between the dynamical theory of critical transitions with the statistical aspects of the MMD. In particular, we show that a formal approximation of the MMD near fast subsystem bifurcation points can be computed to leading-order. In particular, this leading order approximation shows that the MMD depends intricately on the fast-slow systems parameters and one can only expect to extract warning signs under rather stringent conditions. However, the MMD turns out to be an excellent binary classifier to detect the change point induced by the critical transition. We cross-validate our results by numerical simulations for a van der Pol-type model.

nlin.PS↗

arXiv2016

Kernel Methods for the Approximation of Nonlinear Systems

We introduce a data-driven order reduction method for nonlinear control systems, drawing on recent progress in machine learning and statistical dimensionality reduction. The method rests on the assumption that the nonlinear system behaves linearly when lifted into a high (or infinite) dimensional feature space where balanced truncation may be carried out implicitly. This leads to a nonlinear reduction map which can be combined with a representation of the system belonging to a reproducing kernel Hilbert space to give a closed, reduced order dynamical system which captures the essential input-output characteristics of the original model. Empirical simulations illustrating the approach are also provided.

math.OC↗

arXiv2016

Kernel Methods for the Approximation of Some Key Quantities of Nonlinear Systems

We introduce a data-based approach to estimating key quantities which arise in the study of nonlinear control systems and random nonlinear dynamical systems. Our approach hinges on the observation that much of the existing linear theory may be readily extended to nonlinear systems - with a reasonable expectation of success - once the nonlinear system has been mapped into a high or infinite dimensional feature space. In particular, we develop computable, non-parametric estimators approximating controllability and observability energy functions for nonlinear systems, and study the ellipsoids they induce. In all cases the relevant quantities are estimated from simulated or observed data. It is then shown that the controllability energy estimator provides a key means for approximating the invariant measure of an ergodic, stochastically forced nonlinear system.

math.OC↗

arXiv2016

Approximation of Lyapunov Functions from Noisy Data

Methods have previously been developed for the approximation of Lyapunov functions using radial basis functions. However these methods assume that the evolution equations are known. We consider the problem of approximating a given Lyapunov function using radial basis functions where the evolution equations are not known, but we instead have sampled data which is contaminated with noise. We propose an algorithm in which we first approximate the underlying vector field, and use this approximation to then approximate the Lyapunov function. Our approach combines elements of machine learning/statistical learning theory with the existing theory of Lyapunov function approximation. Error estimates are provided for our algorithm.

math.DS↗

arXiv2015

Kernel Methods for Linear Discrete-Time Equations

Methods from learning theory are used in the state space of linear dynamical and control systems in order to estimate the system matrices. An application to stabilization via algebraic Riccati equations is included. The approach is illustrated via a series of numerical examples.

math.DS↗

arXiv2013

A characterization of normal forms for control systems

The study of the behavior of solutions of ODEs often benefits from deciding on a convenient choice of coordinates. This choice of coordinates may be used to "simplify" the functional expressions that appear in the vector field in order that the essential features of the flow of the ODE near a critical point become more evident. In the case of the analysis of an ordinary differential equation in the neighborhood of an equilibrium point, this naturally leads to the consideration of the possibility to remove the maximum number of terms in the Taylor expansion of the vector field up to a given order. This idea was introduced by H. Poincaré and the "simplified" system is called normal form. On another side, even though in many textbook treatments the emphasis is on the reduction of the number of monomials in the Taylor expansion, one of the main reasons for the success of normal forms lies in the fact that it allows to analyze a dynamical system based on a simpler form and a simpler form doesn't necessarily mean to remove the maximum number of terms in the Taylor series expansion. This observation, led to introduce the so-called "inner-product normal forms". They are based on properly choosing an inner product that allows to simplify the computations. This inner-product will characterize the space overwhich one performs the Taylor series expansion. The elements in this space are the ones that characterize the normal form. Our goal in this paper is to generalize such an approach to control systems.

math.OC↗

arXiv2010

Balanced Reduction of Nonlinear Control Systems in Reproducing Kernel Hilbert Space

We introduce a novel data-driven order reduction method for nonlinear control systems, drawing on recent progress in machine learning and statistical dimensionality reduction. The method rests on the assumption that the nonlinear system behaves linearly when lifted into a high (or infinite) dimensional feature space where balanced truncation may be carried out implicitly. This leads to a nonlinear reduction map which can be combined with a representation of the system belonging to a reproducing kernel Hilbert space to give a closed, reduced order dynamical system which captures the essential input-output characteristics of the original model. Empirical simulations illustrating the approach are also provided.

math.OC↗