Faithful-Newton Framework: Bridging Inner and Outer Solvers for Enhanced Optimization

Abstract

Newton-type methods enjoy fast local convergence and strong empirical performance, but achieving global guarantees comparable to first-order methods remains challenging. Even for simple strongly convex problems, no straightforward variant of Newton's method matches the global complexity of gradient descent. While more sophisticated variants can improve iteration complexity, they typically require solving difficult subproblems with high per-iteration costs, leading to worse overall complexity. These limitations stem from treating the subproblem as an afterthought, either as a black box, yielding overly complex and impractical formulations, or in isolation, without regard to its role in advancing the optimization of the main objective. By tightening the integration between the inner iterations of the subproblem solvers and the outer iterations of the optimization algorithm, we introduce simple Newton-type variants, called Faithful-Newton framework, which, in a sense, remain faithful to the overall simplicity of classical Newton's method by retaining simple linear system subproblems. The key conceptual difference, however, is that the quality of the subproblem solution is directly assessed based on its effectiveness in reducing optimality, which in turn enables desirable convergence complexities across a variety of settings. Under standard assumptions, we show that our variants, depending on parameter choices, achieve global superlinear convergence, condition-number-independent linear convergence, and/or local quadratic convergence, even when using inexact Newton steps, for strongly convex problems; and competitive iteration complexity for general convex problems. Numerical experiments further demonstrate that our proposed methods perform competitively compared with several alternative Newton-type approaches.