First-order Optimization Methods with Generalized Smoothness for Low-rank Matrix Factorization

Abstract

This paper studies several first-order methods for low-rank matrix minimization in the Burer--Monteiro form, where the target matrix is represented as the product of a nonnegative factor matrix and its transpose. Such a factorization destroys convexity of the objective function and global Lipschitz continuity of the gradient. We prove that for a quadratic objective function that is strongly convex only on the subspace of symmetric matrices, the Burer-Monteiro factorized objective possesses generalized smoothness with a fractional exponent 2/3. For a quadratic objective, the generalized smoothness property requires positive definiteness of the operator A only on the subspace of symmetric matrices, which relaxes the assumptions of Dragomir et al. On this basis, three methods are proposed: an adaptive gradient descent method with Armijo backtracking; an unconstrained SafeAdam method combining the Adam direction with a descent-angle certificate and a fallback to the normalized gradient; and a certified projected version for the nonnegativity constraint. Global convergence and explicit convergence rates for the proposed methods have been proved. The obtained convergence rate is better than the convergence rate of ordinary unaccelerated gradient descent in the case of a sufficiently large gradient. In the small-gradient regime, the convergence guarantee is of square-root type, and the corresponding complexity scales as the inverse square of the desired accuracy. In the large-gradient regime, the convergence guarantee is of three-quarters type, and the corresponding complexity is proportional to the inverse of the accuracy raised to the power of 4/3. Experiments on SymNMF using the TDT2, Reuters and CBCL datasets, which are standard benchmarks for document and image clustering, show that the methods match or outperform Dyn-NoLips without tuning the relative-smoothness kernel parameters.