The ReLU Matrix Decomposition: NP-Hardness, Ill-Posedness, and ReLU-Rank

Abstract

The ReLU Matrix Decomposition (RMD) approximates a sparse, nonnegative m-by-n matrix X by max(0, WH), where W is an m-by-r matrix, H is an r-by-n matrix, and r is the rank of the decomposition. The most common formulation, referred to as Fro-RMD, minimizes the Frobenius norm of X - max(0, WH) over W and H. We prove that Fro-RMD is NP-hard, even for binary matrices and r = 1; to the best of our knowledge, this is the first complexity result for RMD. We also show that Fro-RMD is ill-posed, in the sense that its infimum may not be attained. We then turn to exact RMD, that is, X = max(0, WH), and define the ReLU-rank of X as the smallest r for which such a decomposition exists. We interpret exact RMDs geometrically as configurations of points and hyperplanes in an r-dimensional space, associated with the rows of W and the columns of H, respectively. Building on this interpretation, we derive upper bounds on the ReLU-rank of diagonal, block-diagonal, and block-sparse matrices and, under an additional structural assumption, improve the best known bound for block-diagonal matrices. Finally, we propose two formulations of exact RMD as feasibility problems: a nonconvex quadratically constrained program, which can be tackled by global solvers such as Gurobi, and a formulation based on the latent model of Saul, (``A nonlinear matrix decomposition for mining the zeros of sparse data', SIMODS, 2022), which we tackle using alternating projections for which we establish local convergence together with asymptotic convergence rates. We compare these two approaches on some numerical examples.