arXiv subjects

Stephen A. Vavasis

Publications and source records attributed to Stephen A. Vavasis.

At least 19 recordsLinked to original sources

arXiv2026

Data-Dependent Complexity of First-Order Methods for Support Vector Machines

Large-scale problems in data science are often modeled with optimization, and the optimization model is usually solved with first-order methods that may converge at a sublinear rate. Therefore, it is of interest to terminate the optimization algorithm as soon as the underlying data science task is accomplished. We analyze the complexity of first-order methods for SVM in terms of the accuracy required to obtain a useful classifier. We derive a simple stopping condition for a perturbed SVM formulation that certifies points as correctly classified by both the current classifier and the optimal classifier, without requiring assumptions on the data. Under a geometric model consisting of two clusters of well-classified points together with noisy observations, we show that certifying a single point guarantees that the current classifier separates all well-classified points provided the number of training samples exceeds a modest threshold. We additionally derive a data-dependent accuracy threshold such that an approximate dual solution within this threshold yields a hyperplane separating the well-classified points. We specialize our theory to FISTA and coordinate descent by deriving computable bounds on the distance to the optimal dual solution, allowing the application of our stopping condition. For FISTA, we obtain data-dependent iteration complexity bounds for satisfying our stopping condition and for separating the well-classified points. Our analysis reveals limitations of the stopping condition used by the well-known software package LIBLINEAR, and we give examples where it is either overly conservative or terminates prematurely with a poor classifier. Numerical experiments show that a FISTA implementation using our stopping condition is competitive with widely used SVM solvers on a range of benchmarks, and demonstrate that our stopping condition provides a robust basis for termination.

math.OC↗

arXiv2026

Range of the displacement operator of PDHG with applications to quadratic and conic programming

Primal-dual hybrid gradient (PDHG) is a first-order method for saddle-point problems and convex programming introduced by Chambolle and Pock. Recently, Applegate et al.\ analyzed the behavior of PDHG when applied to an infeasible or unbounded instance of linear programming, and in particular, showed that PDHG is able to diagnose these conditions. Their analysis hinges on the notion of the infimal displacement vector in the closure of the range of the displacement mapping of the splitting operator that encodes the PDHG algorithm. In this paper, we develop a novel formula for this range using monotone operator theory. The analysis is then specialized to conic programming and further to quadratic programming (QP) and second-order cone programming (SOCP). A consequence of our analysis is that PDHG is able to diagnose infeasible or unbounded instances of QP and of the ellipsoid-separation problem, a subclass of SOCP.

math.OC↗

arXiv2025

Accelerated Proximal Gradient Methods in the affine-quadratic case: Strong convergence and limit identification

Recent works by Bot-Fadili-Nguyen (arXiv:2510.22715) and by Jang-Ryu (arXiv:2510.23513) resolve long-standing iterate convergence questions for accelerated (proximal) gradient methods. In particular, Bot-Fadili-Nguyen prove weak convergence of discrete accelerated gradient descent (AGD) iterates and, crucially, convergence of the accelerated proximal gradient (APG) method in the composite setting, with extensions to infinite-dimensional Hilbert spaces. In parallel, Jang-Ryu establish point convergence for the continuous-time accelerated flow and for discrete AGD in finite dimensions. These results leave open which minimizer is selected by the iterates. We answer this in the affine-quadratic setting: when initialized at the same point, the difference between the proximal gradient (PGM) and APG iterates converges weakly to zero. Consequently, APG converges weakly to the best approximation of the initial point in the solution set. Moreover, under mild assumptions on the parameter sequence, we obtain strong convergence of APG. The result is tight: a two-dimensional example shows that coincidence of the APG and PGM limits is specific to the affine-quadratic regime and does not hold in general.

math.OC↗

arXiv2024

A Primal-Dual Frank-Wolfe Algorithm for Linear Programming

We present two first-order primal-dual algorithms for solving saddle point formulations of linear programs, namely FWLP (Frank-Wolfe Linear Programming) and FWLP-P. The former iteratively applies the Frank-Wolfe algorithm to both the primal and dual of the saddle point formulation of a standard-form LP. The latter is a modification of FWLP in which regularizing perturbations are used in computing the iterates. We show that FWLP-P converges to a primal-dual solution with error $O(1/\sqrt{k})$ after $k$ iterations, while no convergence guarantees are provided for FWLP. We also discuss the advantages of using FWLP and FWLP-P for solving very large LPs. In particular, we argue that only part of the matrix $A$ is needed at each iteration, in contrast to other first-order methods.

math.OC↗

arXiv2023

Accelerated gradient descent: A guaranteed bound for a heuristic restart strategy

The $O(1/k^2)$ convergence rate in function value of accelerated gradient descent is optimal, but there are many modifications that have been used to speed up convergence in practice. Among these modifications are restarts, that is, starting the algorithm with the current iteration being considered as the initial point. We focus on the adaptive restart techniques introduced by O'Donoghue and Candès, specifically their gradient restart strategy. While the gradient restart strategy is a heuristic in general, we prove that applying gradient restarts preserves and in fact improves the $O(1/k^2)$ bound, hence establishing function value convergence, for one-dimensional functions. Applications of our results to separable and nearly separable functions are presented.

math.OC↗

arXiv2023

Computational complexity of decomposing a symmetric matrix as a sum of positive semidefinite and diagonal matrices

We study several variants of decomposing a symmetric matrix into a sum of a low-rank positive semidefinite matrix and a diagonal matrix. Such decompositions have applications in factor analysis and they have been studied for many decades. On the one hand, we prove that when the rank of the positive semidefinite matrix in the decomposition is bounded above by an absolute constant, the problem can be solved in polynomial time. On the other hand, we prove that, in general, these problems as well as their certain approximation versions are all NP-hard. Finally, we prove that many of these low-rank decomposition problems are complete in the first-order theory of the reals; i.e., given any system of polynomial equations, we can write down a low-rank decomposition problem in polynomial time so that the original system has a solution iff our corresponding decomposition problem has a feasible solution of certain (lowest) rank.

math.OC↗

arXiv2020

A termination criterion for stochastic gradient descent for binary classification

We propose a new, simple, and computationally inexpensive termination test for constant step-size stochastic gradient descent (SGD) applied to binary classification on the logistic and hinge loss with homogeneous linear predictors. Our theoretical results support the effectiveness of our stopping criterion when the data is Gaussian distributed. This presence of noise allows for the possibility of non-separable data. We show that our test terminates in a finite number of iterations and when the noise in the data is not too large, the expected classifier at termination nearly minimizes the probability of misclassification. Finally, numerical experiments indicate for both real and synthetic data sets that our termination test exhibits a good degree of predictability on accuracy and running time.

math.OC↗

arXiv2019

Second-order cone interior-point method for quasistatic and moderate dynamic cohesive fracture

Cohesive fracture is among the few techniques able to model complex fracture nucleation and propagation with a sharp (nonsmeared) representation of the crack. Implicit time-stepping schemes are often favored in mechanics due to their ability to take larger time steps in quasistatic and moderate dynamic problems. Furthermore, initially rigid cohesive models are typically preferred when the location of the crack is not known in advance, since initially elastic models artificially lower the material stiffness. It is challenging to include an initially rigid cohesive model in an implicit scheme because the initiation of fracture corresponds to a nondifferentiability of the underlying potential. In this work, an interior-point method is proposed for implicit time stepping of initially rigid cohesive fracture. It uses techniques developed for convex second-order cone programming for the nonconvex problem at hand. The underlying cohesive model is taken from Papoulia (2017) and is based on a nondifferentiable energy function. That previous work proposed an algorithm based on successive smooth approximations to the nondifferential objective for solving the resulting optimization problem. It is argued herein that cone programming can capture the nondifferentiability without smoothing, and the resulting cone formulation is amenable to interior-point algorithms. A further benefit of the formulation is that other conic inequality constraints are straightforward to incorporate. Computational results are provided showing that certain contact constraints can be easily handled and that the method is practical.

math.NA↗

arXiv2017

On the Complexity of Robust PCA and $\ell_1$-norm Low-Rank Matrix Approximation

The low-rank matrix approximation problem with respect to the component-wise $\ell_1$-norm ($\ell_1$-LRA), which is closely related to robust principal component analysis (PCA), has become a very popular tool in data mining and machine learning. Robust PCA aims at recovering a low-rank matrix that was perturbed with sparse noise, with applications for example in foreground-background video separation. Although $\ell_1$-LRA is strongly believed to be NP-hard, there is, to the best of our knowledge, no formal proof of this fact. In this paper, we prove that $\ell_1$-LRA is NP-hard, already in the rank-one case, using a reduction from MAX CUT. Our derivations draw interesting connections between $\ell_1$-LRA and several other well-known problems, namely, robust PCA, $\ell_0$-LRA, binary matrix factorization, a particular densest bipartite subgraph problem, the computation of the cut norm of $\{-1,+1\}$ matrices, and the discrete basis problem, which we all prove to be NP-hard.

cs.LG↗

arXiv2016

A unified convergence bound for conjugate gradient and accelerated gradient

Nesterov's accelerated gradient method for minimizing a smooth strongly convex function $f$ is known to reduce $f(\x_k)-f(\x^*)$ by a factor of $\eps\in(0,1)$ after $k\ge O(\sqrt{L/\ell}\log(1/\eps))$ iterations, where $\ell,L$ are the two parameters of smooth strong convexity. Furthermore, it is known that this is the best possible complexity in the function-gradient oracle model of computation. The method of linear conjugate gradients (CG) also satisfies the same complexity bound in the special case of strongly convex quadratic functions, but in this special case it is faster than the accelerated gradient method. Despite similarities in the algorithms and their asymptotic convergence rates, the conventional analyses of the two methods are nearly disjoint. The purpose of this note is provide a single quantity that decreases on every step at the correct rate for both algorithms. Our unified bound is based on a potential similar to the potential in Nesterov's original analysis. As a side benefit of this analysis, we provide a direct proof that conjugate gradient converges in $O(\sqrt{L/\ell}\log(1/\eps))$ iterations. In contrast, the traditional indirect proof first establishes this result for the Chebyshev algorithm, and then relies on optimality of conjugate gradient to show that its iterates are at least as good as Chebyshev iterates. To the best of our knowledge, ours is the first direct proof of the convergence rate of linear conjugate gradient in the literature.

math.OC↗

arXiv2014

Semidefinite Programming Based Preconditioning for More Robust Near-Separable Nonnegative Matrix Factorization

Nonnegative matrix factorization (NMF) under the separability assumption can provably be solved efficiently, even in the presence of noise, and has been shown to be a powerful technique in document classification and hyperspectral unmixing. This problem is referred to as near-separable NMF and requires that there exists a cone spanned by a small subset of the columns of the input nonnegative matrix approximately containing all columns. In this paper, we propose a preconditioning based on semidefinite programming making the input matrix well-conditioned. This in turn can improve significantly the performance of near-separable NMF algorithms which is illustrated on the popular successive projection algorithm (SPA). The new preconditioned SPA is provably more robust to noise, and outperforms SPA on several synthetic data sets. We also show how an active-set method allow us to apply the preconditioning on large-scale real-world hyperspectral images.

stat.ML↗

arXiv2013

Convex optimization for the planted k-disjoint-clique problem

We consider the k-disjoint-clique problem. The input is an undirected graph G in which the nodes represent data items, and edges indicate a similarity between the corresponding items. The problem is to find within the graph k disjoint cliques that cover the maximum number of nodes of G. This problem may be understood as a general way to pose the classical `clustering' problem. In clustering, one is given data items and a distance function, and one wishes to partition the data into disjoint clusters of data items, such that the items in each cluster are close to each other. Our formulation additionally allows `noise' nodes to be present in the input data that are not part of any of the cliques. The k-disjoint-clique problem is NP-hard, but we show that a convex relaxation can solve it in polynomial time for input instances constructed in a certain way. The input instances for which our algorithm finds the optimal solution consist of k disjoint large cliques (called `planted cliques') that are then obscured by noise edges and noise nodes inserted either at random or by an adversary.

math.OC↗

arXiv2013

Fast and Robust Recursive Algorithms for Separable Nonnegative Matrix Factorization

In this paper, we study the nonnegative matrix factorization problem under the separability assumption (that is, there exists a cone spanned by a small subset of the columns of the input nonnegative data matrix containing all columns), which is equivalent to the hyperspectral unmixing problem under the linear mixing model and the pure-pixel assumption. We present a family of fast recursive algorithms, and prove they are robust under any small perturbations of the input data matrix. This family generalizes several existing hyperspectral unmixing algorithms and hence provides for the first time a theoretical justification of their better practical performance.

stat.ML↗

arXiv2011

A Robust Solution Procedure for Hyperelastic Solids with Large Boundary Deformation

Compressible Mooney-Rivlin theory has been used to model hyperelastic solids, such as rubber and porous polymers, and more recently for the modeling of soft tissues for biomedical tissues, undergoing large elastic deformations. We propose a solution procedure for Lagrangian finite element discretization of a static nonlinear compressible Mooney-Rivlin hyperelastic solid. We consider the case in which the boundary condition is a large prescribed deformation, so that mesh tangling becomes an obstacle for straightforward algorithms. Our solution procedure involves a largely geometric procedure to untangle the mesh: solution of a sequence of linear systems to obtain initial guesses for interior nodal positions for which no element is inverted. After the mesh is untangled, we take Newton iterations to converge to a mechanical equilibrium. The Newton iterations are safeguarded by a line search similar to one used in optimization. Our computational results indicate that the algorithm is up to 70 times faster than a straightforward Newton continuation procedure and is also more robust (i.e., able to tolerate much larger deformations). For a few extremely large deformations, the deformed mesh could only be computed through the use of an expensive Newton continuation method while using a tight convergence tolerance and taking very small steps.

math.NA↗

arXiv2011

Analysis of and workarounds for element reversal for a finite element-based algorithm for warping triangular and tetrahedral meshes

We consider an algorithm called FEMWARP for warping triangular and tetrahedral finite element meshes that computes the warping using the finite element method itself. The algorithm takes as input a two- or three-dimensional domain defined by a boundary mesh (segments in one dimension or triangles in two dimensions) that has a volume mesh (triangles in two dimensions or tetrahedra in three dimensions) in its interior. It also takes as input a prescribed movement of the boundary mesh. It computes as output updated positions of the vertices of the volume mesh. The first step of the algorithm is to determine from the initial mesh a set of local weights for each interior vertex that describes each interior vertex in terms of the positions of its neighbors. These weights are computed using a finite element stiffness matrix. After a boundary transformation is applied, a linear system of equations based upon the weights is solved to determine the final positions of the interior vertices. The FEMWARP algorithm has been considered in the previous literature (e.g., in a 2001 paper by Baker). FEMWARP has been succesful in computing deformed meshes for certain applications. However, sometimes FEMWARP reverses elements; this is our main concern in this paper. We analyze the causes for this undesirable behavior and propose several techniques to make the method more robust against reversals. The most successful of the proposed methods includes combining FEMWARP with an optimization-based untangler.

math.NA↗

arXiv2010

Finding approximately rank-one submatrices with the nuclear norm and l1 norm

We propose a convex optimization formulation with the nuclear norm and $\ell_1$-norm to find a large approximately rank-one submatrix of a given nonnegative matrix. We develop optimality conditions for the formulation and characterize the properties of the optimal solutions. We establish conditions under which the optimal solution of the convex formulation has a specific sparse structure. Finally, we show that, under certain hypotheses, with high probability, the approach can recover the rank-one submatrix even when it is corrupted with random noise and inserted as a submatrix into a much larger random noise matrix.

math.OC↗

arXiv2009

A condition number analysis of an algorithm for solving a system of polynomial equations with one degree of freedom

This article considers the problem of solving a system of $n$ real polynomial equations in $n+1$ variables. We propose an algorithm based on Newton's method and subdivision for this problem. Our algorithm is intended only for nondegenerate cases, in which case the solution is a 1-dimensional curve. Our first main contribution is a definition of a condition number measuring reciprocal distance to degeneracy that can distinguish poor and well conditioned instances of this problem. (Degenerate problems would be infinitely ill conditioned in our framework.) Our second contribution, which is the main novelty of our algorithm, is an analysis showing that its running time is bounded in terms of the condition number of the problem instance as well as $n$ and the polynomial degrees.

cs.CG↗

arXiv2009

Properties of polynomial bases used in a line-surface intersection algorithm

In [5], Srijuntongsiri and Vavasis propose the "Kantorovich-Test Subdivision algorithm", or KTS, which is an algorithm for finding all zeros of a polynomial system in a bounded region of the plane. This algorithm can be used to find the intersections between a line and a surface. The main features of KTS are that it can operate on polynomials represented in any basis that satisfies certain conditions and that its efficiency has an upper bound that depends only on the conditioning of the problem and the choice of the basis representing the polynomial system. This article explores in detail the dependence of the efficiency of the KTS algorithm on the choice of basis. Three bases are considered: the power, the Bernstein, and the Chebyshev bases. These three bases satisfy the basis properties required by KTS. Theoretically, Chebyshev case has the smallest upper bound on its running time. The computational results, however, do not show that Chebyshev case performs better than the other two.

math.NA↗