Several Accelerated and Stable Pseudo-Energy-Dissipative Lagrange Multiplier Methods based on Second-order Flow and Variable Splitting

Abstract

Based on second-order inertial dynamics, we develop two accelerated Lagrange multiplier (LM)-based optimization methods: Linear LM-based second-order flow method and linear LM-based variable and operator splitting method for unconstrained non-convex optimization problems. Relaxed and adaptive variants of both methods are proposed to prevent degeneration of the Lagrange multiplier and to enable automatic adjustment of the time step size. Within the novel second-order LM-based optimization framework, the associated pseudo-energy functional is shown to dissipate monotonically along the iterative trajectory, and convergence of the generated iterates to stationary points is rigorously established. Extensive numerical experiments on function optimization and partial differential equation benchmark problems, including applications as optimizers for physics-informed neural networks (PINNs) and deep operator networks (DeepONets), demonstrate that the proposed methods effectively escape local minima, achieve high accuracy, and exhibit strong robustness with respect to the choice of the initial learning rate during neural network training.