arXiv subjects

Mathieu Laurière

Publications and source records attributed to Mathieu Laurière.

At least 37 records · Page 2Linked to original sources

arXiv2026

Controlling Exploration-Exploitation in GFlowNets via Markov Chain Perspectives

Generative Flow Network (GFlowNet) objectives implicitly fix an equal mixing of forward and backward policies, potentially constraining the exploration-exploitation trade-off during training. By further exploring the link between GFlowNets and Markov chains, we establish an equivalence between GFlowNet objectives and Markov chain reversibility, thereby revealing the origin of such constraints, and provide a framework for adapting Markov chain properties to GFlowNets. Building on these theoretical findings, we propose $α$-GFNs, which generalize the mixing via a tunable parameter $α$. This generalization enables direct control over exploration-exploitation dynamics to enhance mode discovery capabilities, while ensuring convergence to unique flows. Across various benchmarks, including Set, Bit Sequence, and Molecule Generation, $α$-GFN objectives consistently outperform previous GFlowNet objectives, achieving up to a $10 \times$ increase in the number of discovered modes.

cs.AI↗

arXiv2026

Bench-MFG: A Benchmark Suite for Learning in Stationary Mean Field Games

The intersection of Mean Field Games (MFGs) and Reinforcement Learning (RL) has fostered a growing family of algorithms designed to solve large-scale multi-agent systems. However, the field currently lacks a standardized evaluation protocol, forcing researchers to rely on bespoke, isolated, and often simplistic environments. This fragmentation makes it difficult to assess the robustness, generalization, and failure modes of emerging methods. To address this gap, we propose a comprehensive benchmark suite for MFGs (Bench-MFG), focusing on the discrete-time, discrete-space, stationary setting for the sake of clarity. We introduce a taxonomy of problem classes, ranging from no-interaction and monotone games to potential and dynamics-coupled games, and provide prototypical environments for each. Furthermore, we propose MF-Garnets, a method for generating random MFG instances to facilitate rigorous statistical testing. We benchmark a variety of learning algorithms across these environments, including a novel black-box approach (MF-PSO) for exploitability minimization. Based on our extensive empirical results, we propose guidelines to standardize future experimental comparisons. Code available at \href{https://github.com/lorenzomagnino/Bench-MFG}{https://github.com/lorenzomagnino/Bench-MFG}.

cs.LG↗

arXiv2026

Operator Learning for Families of Finite-State Mean-Field Games

Finite-state mean-field games (MFGs) arise as limits of large interacting particle systems and are governed by an MFG system, a coupled forward-backward differential equation consisting of a forward Kolmogorov-Fokker-Planck (KFP) equation describing the population distribution and a backward Hamilton-Jacobi-Bellman (HJB) equation defining the value function. Solving MFG systems efficiently is challenging, with the structure of each system depending on an initial distribution of players and the terminal cost of the game. We propose an operator learning framework that solves parametric families of MFGs, enabling generalization without retraining for new initial distributions and terminal costs. We provide theoretical guarantees on the approximation error, parametric complexity, and generalization performance of our method, based on a novel regularity result for an appropriately defined flow map corresponding to an MFG system. We demonstrate empirically that our framework achieves accurate approximation for two representative instances of MFGs: a cybersecurity example and a high-dimensional quadratic model commonly used as a benchmark for numerical methods for MFGs.

math.OC↗

arXiv2026

Dynamic Programming Principle and Stabilization for Mean-Field Quantum Filtering Systems

Working within the quantum filtering framework, we establish a dynamic programming principle in an infinite-dimensional setting by embedding the state space into the Hilbert-Schmidt space. We then study a stabilization problem for continuously monitored Ising-coupled qubits and, in the mean-field limit, demonstrate quantum state reduction together with exponential convergence toward prescribed eigenstates under suitable feedback laws.

quant-ph↗

arXiv2026

Clustering in Deep Stochastic Transformers

Transformers have revolutionized deep learning across various domains but understanding the precise token dynamics remains a theoretical challenge. Existing theories of deep Transformers with layer normalization typically predict that tokens cluster to a single point; however, these results rely on deterministic weight assumptions, which fail to capture the standard initialization scheme in Transformers. In this work, we show that accounting for the intrinsic stochasticity of random initialization alters this picture. More precisely, we analyze deep Transformers where noise arises from the random initialization of value matrices. Under diffusion scaling and token-wise RMS normalization, we prove that, as the number of Transformer layers goes to infinity, the discrete token dynamics converge to an interacting-particle system on the sphere where tokens are driven by a \emph{common} matrix-valued Brownian noise. In this limit, we show that initialization noise prevents the collapse to a single cluster predicted by deterministic models. For two tokens, we prove a phase transition governed by the interaction strength and the token dimension: unlike deterministic attention flows, antipodal configurations become attracting with positive probability. Numerical experiments confirm the predicted transition, reveal that antipodal formations persist for more than two tokens, and demonstrate that suppressing the intrinsic noise degrades accuracy.

stat.ML↗

arXiv2025

Deep Learning for the Multiple Optimal Stopping Problem

This paper presents a novel deep learning framework for solving multiple optimal stopping problems in high dimensions. While deep learning has recently shown promise for single stopping problems, the multiple exercise case involves complex recursive dependencies that remain challenging. We address this by combining the Dynamic Programming Principle with neural network approximation of the value function. Unlike policy-search methods, our algorithm explicitly learns the value surface. We first consider the discrete-time problem and analyze neural network training error. We then turn to continuous problems and analyze the additional error due to the discretization of the underlying stochastic processes. Numerical experiments on high-dimensional American basket options and nonlinear utility maximization demonstrate that our method provides an efficient and scalable method for the multiple optimal stopping problem.

math.OC↗

arXiv2025

Probabilistic Analysis of Graphon Mean Field Control

Motivated by recent interest in graphon mean field games and their applications, this paper provides a comprehensive probabilistic analysis of graphon mean field control (GMFC) problems, where the controlled dynamics are governed by a graphon mean field stochastic differential equation with heterogeneous mean field interactions. We formulate the GMFC problem with general graphon mean field dependence and establish the existence and uniqueness of the associated graphon mean field forward-backward stochastic differential equations (FBSDEs). We then derive a version of the Pontryagin stochastic maximum principle tailored to GMFC problems. Furthermore, we analyze the solvability of the GMFC problem for linear dynamics and study the continuity and stability of the graphon mean field FBSDEs under the optimal control profile. Finally, we show that the solution to the GMFC problem provides an approximately optimal solution for large systems with heterogeneous mean field interactions, based on a propagation of chaos result.

math.OC↗

arXiv2025

Deep Signature Approach for McKean-Vlasov FBSDEs in a Random Environment

Mean-field games with common noise provide a powerful framework for modeling the collective behavior of large populations subject to shared randomness, such as systemic risk in finance or environmental shocks in economics. These problems can be reformulated as McKean-Vlasov forward-backward stochastic differential equations (MV-FBSDEs) in a random environment, where the coefficients depend on the conditional law of the state given the common noise. Existing numerical methods, however, are largely limited to cases where interactions depend only on expectations or low-order moments, and therefore cannot address the general setting of full distributional dependence. In this work, we introduce a deep learning-based algorithm for solving MV-FBSDEs with common noise and general mean-field interactions. Building on fictitious play, our method iteratively solves conditional FBSDEs with fixed distributions, where the conditional law is efficiently represented using signatures, and then updates the distribution through supervised learning. Deep neural networks are employed both to solve the conditional FBSDEs and to approximate the distribution-dependent coefficients, enabling scalability to high-dimensional problems. Under suitable assumptions, we establish convergence in terms of the fictitious play iterations, with error controlled by the supervised learning step. Numerical experiments, including a distribution-dependent mean-field game with common noise, demonstrate the effectiveness of the proposed approach.

math.OC↗

arXiv2025

Convergence of Actor-Critic Learning for Mean Field Games and Mean Field Control in Continuous Spaces

We establish the convergence of the deep actor-critic reinforcement learning algorithm presented in [Angiuli et al., 2023a] in the setting of continuous state and action spaces with an infinite discrete-time horizon. This algorithm provides solutions to Mean Field Game (MFG) or Mean Field Control (MFC) problems depending on the ratio between two learning rates: one for the value function and the other for the mean field term. In the MFC case, to rigorously identify the limit, we introduce a discretization of the state and action spaces, following the approach used in the finite-space case in [Angiuli et al., 2023b]. The convergence proofs rely on a generalization of the two-timescale framework introduced in [Borkar, 1997]. We further extend our convergence results to Mean Field Control Games, which involve locally cooperative and globally competitive populations. Finally, we present numerical experiments for linear-quadratic problems in one and two dimensions, for which explicit solutions are available.

math.OC↗

arXiv2025

An Overview of Some Extensions of Mean Field Games beyond Perfect Homogeneity and Anonymity

The mean field games (MFG) paradigm was introduced to provide tractable approximations of games involving very large populations. The theory typically rests on two key assumptions: homogeneity, meaning that all players share the same dynamics and cost functions, and anonymity, meaning that each player interacts with others only through their empirical distribution. While these assumptions simplify the analysis, they can be restrictive for many applications. Fortunately, several extensions of the standard MFG framework that relax these assumptions have been developed in the literature. The purpose of these notes is to offer a pedagogical introduction to such models. In particular, we discuss multi-population MFGs, graphon MFGs, major-minor MFGs, and Stackelberg MFGs, as well as variants involving cooperative players.

math.OC↗

arXiv2025

Robust mean-field control under common noise uncertainty

We propose and analyze a framework for discrete-time robust mean-field control problems under common noise uncertainty. In this framework, the mean-field interaction describes the collective behavior of infinitely many cooperative agents' state and action, while the common noise -- a random disturbance affecting all agents' state dynamics -- is uncertain. A social planner optimizes over open-loop controls on an infinite horizon to maximize the representative agent's worst-case expected reward, where worst-case corresponds to the most adverse probability measure among all candidates inducing the unknown true law of the common noise process. We refer to this optimization as a robust mean-field control problem under common noise uncertainty. We first show that this problem arises as the asymptotic limit of a cooperative $N$-agent robust optimization problem, commonly known as propagation of chaos. We then prove the existence of an optimal open-loop control by linking the robust mean field control problem to a lifted robust Markov decision problem on the space of probability measures and by establishing the dynamic programming principle and Bellman--Isaac fixed point theorem for the lifted robust Markov decision problem. Finally, we complement our theoretical results with numerical experiments motivated by distribution planning and systemic risk in finance, highlighting the advantages of accounting for common noise uncertainty.

math.OC↗

arXiv2025

Solving Continuous Mean Field Games: Deep Reinforcement Learning for Non-Stationary Dynamics

Mean field games (MFGs) have emerged as a powerful framework for modeling interactions in large-scale multi-agent systems. Despite recent advancements in reinforcement learning (RL) for MFGs, existing methods are typically limited to finite spaces or stationary models, hindering their applicability to real-world problems. This paper introduces a novel deep reinforcement learning (DRL) algorithm specifically designed for non-stationary continuous MFGs. The proposed approach builds upon a Fictitious Play (FP) methodology, leveraging DRL for best-response computation and supervised learning for average policy representation. Furthermore, it learns a representation of the time-dependent population distribution using a Conditional Normalizing Flow. To validate the effectiveness of our method, we evaluate it on three different examples of increasing complexity. By addressing critical limitations in scalability and density approximation, this work represents a significant advancement in applying DRL techniques to complex MFG problems, bringing the field closer to real-world multi-agent systems.

cs.LG↗

arXiv2025

Iterative Schemes for Markov Perfect Equilibria

We study Markov perfect equilibria in continuous-time dynamic games with finitely many symmetric players. The corresponding Nash system reduces to the Nash-Lasry-Lions equation for the common value function, also known as the master equation in the mean-field setting. In the finite-state space problems we consider, this equation becomes a nonlinear ordinary differential equation admitting a unique classical solution. Leveraging this uniqueness, we prove the convergence of both Picard and weighted Picard iterations, yielding efficient computational methods. Numerical experiments confirm the effectiveness of algorithms based on this approach.

math.OC↗

arXiv2025

Reinforcement Learning for Finite Space Mean-Field Type Games

Mean field type games (MFTGs) describe Nash equilibria between large coalitions: each coalition consists of a continuum of cooperative agents who maximize the average reward of their coalition while interacting non-cooperatively with a finite number of other coalitions. Although the theory has been extensively developed, we are still lacking efficient and scalable computational methods. Here, we develop reinforcement learning methods for such games in a finite space setting with general dynamics and reward functions. We start by proving that the MFTG solution yields approximate Nash equilibria in finite-size coalition games. We then propose two algorithms. The first is based on the quantization of mean-field spaces and Nash Q-learning. We provide convergence and stability analysis. We then propose a deep reinforcement learning algorithm, which can scale to larger spaces. Numerical experiments in 4 environments with mean-field distributions of dimension up to $200$ show the scalability and efficiency of the proposed method.

cs.GT↗

arXiv2025

Robust Cooperative Multi-Agent Reinforcement Learning:A Mean-Field Type Game Perspective

In this paper, we study the problem of robust cooperative multi-agent reinforcement learning (RL) where a large number of cooperative agents with distributed information aim to learn policies in the presence of \emph{stochastic} and \emph{non-stochastic} uncertainties whose distributions are respectively known and unknown. Focusing on policy optimization that accounts for both types of uncertainties, we formulate the problem in a worst-case (minimax) framework, which is is intractable in general. Thus, we focus on the Linear Quadratic setting to derive benchmark solutions. First, since no standard theory exists for this problem due to the distributed information structure, we utilize the Mean-Field Type Game (MFTG) paradigm to establish guarantees on the solution quality in the sense of achieved Nash equilibrium of the MFTG. This in turn allows us to compare the performance against the corresponding original robust multi-agent control problem. Then, we propose a Receding-horizon Gradient Descent Ascent RL algorithm to find the MFTG Nash equilibrium and we prove a non-asymptotic rate of convergence. Finally, we provide numerical experiments to demonstrate the efficacy of our approach relative to a baseline algorithm.

cs.MA↗

arXiv2025

Learning to Stop: Deep Learning for Mean Field Optimal Stopping

Optimal stopping is a fundamental problem in optimization with applications in risk management, finance, robotics, and machine learning. We extend the standard framework to a multi-agent setting, named multi-agent optimal stopping (MAOS), where agents cooperate to make optimal stopping decisions in a finite-space, discrete-time environment. Since solving MAOS becomes computationally prohibitive as the number of agents is very large, we study the mean-field optimal stopping (MFOS) problem, obtained as the number of agents tends to infinity. We establish that MFOS provides a good approximation to MAOS and prove a dynamic programming principle (DPP) based on mean-field control theory. We then propose two deep learning approaches: one that learns optimal stopping decisions by simulating full trajectories and another that leverages the DPP to compute the value function and to learn the optimal stopping rule using backward induction. Both methods train neural networks to approximate optimal stopping policies. We demonstrate the effectiveness and the scalability of our work through numerical experiments on 6 different problems in spatial dimension up to 300. To the best of our knowledge, this is the first work to formalize and computationally solve MFOS in discrete time and finite space, opening new directions for scalable MAOS methods.

math.OC↗

arXiv2025

Finite-Sample Convergence Bounds for Trust Region Policy Optimization in Mean-Field Games

We introduce Mean-Field Trust Region Policy Optimization (MF-TRPO), a novel algorithm designed to compute approximate Nash equilibria for ergodic Mean-Field Games (MFG) in finite state-action spaces. Building on the well-established performance of TRPO in the reinforcement learning (RL) setting, we extend its methodology to the MFG framework, leveraging its stability and robustness in policy optimization. Under standard assumptions in the MFG literature, we provide a rigorous analysis of MF-TRPO, establishing theoretical guarantees on its convergence. Our results cover both the exact formulation of the algorithm and its sample-based counterpart, where we derive high-probability guarantees and finite sample complexity. This work advances MFG optimization by bridging RL techniques with mean-field decision-making, offering a theoretically grounded approach to solving complex multi-agent problems.

stat.ML↗

arXiv2025

An Efficient On-Policy Deep Learning Framework for Stochastic Optimal Control

We present a novel on-policy algorithm for solving stochastic optimal control (SOC) problems. By leveraging the Girsanov theorem, our method directly computes on-policy gradients of the SOC objective without expensive backpropagation through stochastic differential equations or adjoint problem solutions. This approach significantly accelerates the optimization of neural network control policies while scaling efficiently to high-dimensional problems and long time horizons. We evaluate our method on classical SOC benchmarks as well as applications to sampling from unnormalized distributions via Schrödinger-Föllmer processes and fine-tuning pre-trained diffusion models. Experimental results demonstrate substantial improvements in both computational speed and memory efficiency compared to existing approaches.

cs.LG↗