arXiv subjects

Andy Yat-Ming Cheung

Publications and source records attributed to Andy Yat-Ming Cheung.

2 recordsLinked to original sources

arXiv2026

Randomized Submanifold Subgradient Method for Optimization over Stiefel Manifolds

Optimization over the Stiefel manifold is a fundamental computational problem in many scientific and engineering applications. Despite considerable research effort, high-dimensional optimization problems over the Stiefel manifold remain challenging, particularly when the objective function is nonsmooth. In this paper, we propose a novel coordinate-type algorithm, named randomized submanifold subgradient method (RSSM), for minimizing a possibly nonsmooth weakly convex function over the Stiefel manifold and study its convergence behavior. In contrast to existing algorithms, RSSM operates on ``coordinate blocks'' defined by subsets of the columns of the matrix variable. Similar to coordinate-type algorithms in the Euclidean setting, RSSM exhibits low per-iteration cost and is suitable for high-dimensional problems. We prove that RSSM has an iteration complexity of $\mathcal O(\ell^{2/3}\eps^{-4})$ for driving a natural stationarity measure below $\eps$ in expectation, where $\ell$ represents the number of blocks. We also establish the almost-sure convergence of RSSM and provide the almost-sure asymptotic convergence rate. To the best of our knowledge, this is the first convergence guarantee for coordinate-type algorithms for nonsmooth optimization over the Stiefel manifold. Lastly, we present numerical results on robust subspace recovery, orthogonal dictionary learning, and sparse principal component analysis to demonstrate the viability of our proposed method.

math.OC↗

arXiv2026

Exploring Diverse Generation Paths via Inference-time Stiefel Activation Steering

Language models often default to a narrow set of high-probability outputs, leaving their generation paths homogeneous and prone to mode collapse. Sampling-based strategies inject randomness but still struggle to guarantee diversity across multiple concurrent generation runs. We address this limitation by introducing STARS ($\textbf{St}$iefel-based $\textbf{A}$ctivation Steering for Diverse $\textbf{R}$ea$\textbf{S}$oning), a training-free, inference-time intervention method that transforms activation steering into an exploration engine. At each token, STARS collects the hidden activations of concurrent generation runs and optimizes multiple additive steering directions jointly on the Stiefel manifold. STARS maximizes the geometric volume of the steered activations, while the Stiefel manifold induces orthogonality of the steering interventions. This formulation explicitly promotes divergent activation vectors of concurrent generation runs, and implicitly promotes divergent generation trajectories. This manifold optimization formulation can be solved using a Riemannian gradient descent algorithm with convergence guarantees, but this algorithm is too time-consuming for real-time inference. To guarantee low latency, we further design a lightweight one-step update with an aggressive, closed-form stepsize. For test case generation and scientific discovery benchmarks, STARS consistently outperforms standard sampling methods, achieving greater diversity without sacrificing qualitative performance.

cs.LG↗