Complexity of Lasso with Normalized Data, Geometry and Combinatorics

Abstract

We observe and prove that the complexity of lasso for normalized data is smaller than for nonnormalized ones by relating this question to extremal combinatorics and algebraic graph theory. We employ a geometric approach to the lasso as a study of the tangency of the level sets of the least square objective function with the polyhedral boundary sets $B(t)$ of the parameters in $\mathbb R^p$ with the $\ell_1$ norm equal to $t$. We geometrically derive closed exact formulae for the solution of the lasso under the full rank assumption. We establish important general properties of the solutions of the lasso, which are known to be represented as a simple polygonal chain in $\mathbb{R}^p$. Starting from $p=2$ and $p=3$, we show a striking difference in the maximal number of $p$-dimensional orthants a polygonal chain of a lasso solution can intersect in the case of normalized data vs. nonnormalized data. We prove that in the normalized case, the number $h_{p,2}$ is a general upper bound for the number of segments of a lasso solution intersecting a $p$-dimensional orthant, where $h_{p,r}$ is the maximal number of binary words of length $p$ such that every two words match at least at $r$ spots, $r\le p$. We prove, using spectral graph theory, that $h_{p,2}=2^{p-1}-\binom{p}{p/2}/2$ for $p$ even and $h_{p,2}=2^{p-1}-\binom{p-1}{(p-1)/2}$ for $p$ odd. It was known that for general data the sharp estimate for that number is $2^{p-1}$, which we identify with $h_{p,1}$. We also find an upper bound for the total number of segments of a lasso solution with normalized data in dimension $p$, e.g. $\hat N(2\ell)=\frac{3^{2\ell}+5}{2} - 4\ell\, {}_3F_2\!\left( \begin{matrix} \frac{1}{2},\;1-\ell,\;\frac{1}{2}-\ell\\[1mm] \frac{3}{2},\;1 \end{matrix} ;4 \right)$, that is significantly less than $(3^p+1)/2$, a well-known sharp upper bound for the nonnormalized case.