Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.16923

Minimax Optimality of Shampoo for Nonstationary Stochastic Optimization

Abir Sarkar·Nirmalya Mandal·Martin T. Wells
View original source ↗Read source document ↗Export metadata ↓

Abstract

We develop a comprehensive theoretical analysis of Matrix Shampoo for tracking time-varying optima in nonstationary stochastic optimization. We derive finite-horizon high-probability bounds for average tracking error that disentangle initialization, optimizer drift, momentum-induced drift penalty, stochastic noise, and adaptive preconditioning, with explicit dependence on momentum parameter \(β_1\) and stabilization parameter \(\varepsilon\). These bounds reveal an explicit drift--noise and stability--geometry tradeoffs: noise-dominated regimes favor larger $β_1$, which provides stronger first-moment averaging and greater suppression of gradient noise, whereas drift-dominated regimes favor smaller $β_1$, allowing faster adaptation to moving optimizer. Larger $\varepsilon$ reduces the preconditioner-perturbation term while weakening Shampoo's ability to exploit matrix geometry, thereby reducing adaptivity. We establish a matching minimax lower bound under squared path-variation constraint, proving the information-theoretic optimality of Shampoo over the class of causal optimization procedures. Under a constant stepsize, we establish high-probability bounds for the average projected stationarity gap along general nonstationary nonconvex objectives. We validate our theoretical findings on both synthetic and multiple real-world problems.

Explore related subjects

stat.MLcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Abir Sarkar, Nirmalya Mandal, Martin T. Wells. 2026-10-12. Minimax Optimality of Shampoo for Nonstationary Stochastic Optimization. https://arxiv.org/abs/2610.16923

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Bayesian Nonparametric Thompson Sampling for Multi-Armed Bandits

In stochastic multi-armed bandits, rewards are generated by an unknown distribution; the world's reward uncertainty induces the exploration-exploitation trade-off. Thompson sampling (TS) addresses this trade-off elegantly, but requires knowing the reward model's parametric form to sample from its posterior and compute per-arm expected rewards ---which limits its applicability as it demands per-scenario model specification and hyperparameter tuning. We here adopt Bayesian nonparametric (BNP) mixture models to remove this requirement, extending TS to arbitrary reward distributions. We model each arm's unknown reward distribution with an independent BNP Gaussian mixture, which can approximate any continuous reward density, and adapts its complexity to sequentially observed data. Embedding the BNP mixture model into TS yields Nonparametric TS, an algorithm that sequentially learns the reward model that best approximates the true, yet unknown, per-arm reward distribution, avoiding case-by-case model specification and hyperparameter tuning. We empirically demonstrate that ($i$) Nonparametric TS matches the regret performance of Oracle TS policies that know the true reward model; ($ii$) it favorably compares (in cumulative regret and volatility) to a suite of state-of-the-art alternatives across multimodal, outlier-contaminated, and non-exponential-family reward distributions; and ($iii$) its Bayesian nonparametric inference remains computationally practical and accurate for regret minimization.

stat.ML↗
arXiv2026

OSCAR: Order-aware Sensitivity Correction for AI Rankings

LLM judges evaluate candidate language models through pairwise comparisons of their responses. Judge-aware Bradley--Terry models aggregate pairwise comparisons by jointly estimating candidate scores and judge-specific sensitivities (how strongly each judge responds to quality differences), often interpreted as reliability. We show that this reading fails when judges prefer a display position. Even under balanced display orders or swap-and-average, omitting position attenuates fitted judge sensitivity by approximately the factor $1-D^2$ near equal quality, where $D$ is the judge's net first-slot preference between equally good responses. We propose OSCAR, which jointly estimates candidate scores, judge sensitivities and judge-specific position, length and family effects, and provides confidence intervals for pairwise win probabilities that account for dependence within prompts. On held-out data, observed attenuation closely matches the prediction $1-D^2$ (correlation of log ratios 0.99). Across four datasets, position adjustment lowers held-out log loss by 14-23% and changes which judges would be dropped for low sensitivity on three datasets. On MT-Bench, it ranks GPT-4 first instead of claude-v1, matching human votes. Simulations with known parameters show improved recovery of judge sensitivities and candidate rankings, with 94.4-95.2% confidence interval coverage. Low estimated sensitivity alone is therefore not evidence of an unreliable judge.

stat.ML↗
arXiv2026

PICPIs: Prediction-Interval-Conditional Prediction Intervals

A classical question in statistics is which observable quantities to condition on when drawing inferences about unobservable targets. For conformal prediction in nonparametric uncertainty quantification, standard marginal validity offers limited resolution at the prediction values on which decisions are based, and fully conditional guarantees with respect to the covariates are provably unattainable. We address this gap by introducing a prediction-based conditioning framework that we refer to as Prediction-Interval-Conditional Prediction Intervals (PICPIs). Formally, a PICPI is an interval $I$ satisfying a self-consistency condition: $$\mathbb{E} [Y \mid p(X) \in I] \in I,$$ for predictive model $p$, contextual covariate $X$, and outcome $Y$. Thus, an interval simultaneously defines a stratum of prediction values and certifies that the mean outcome in that stratum lies in the same interval. This self-consistency condition yields data-adaptive strata without altering the original prediction. Such intervals can be constructed using practical algorithms. Under regularity of the prediction distribution, the constructed intervals cover all but an arbitrarily small fraction of prediction values and have widths that decrease at rate $n^{-1/3}$, up to logarithmic factors and the prediction error. Moreover, identifying these locally calibrated intervals can, in turn, inform downstream decision-making. We derive inference procedures for PICPIs in probabilistic prediction and multi-class classification, accompanied by theoretical guarantees. Empirical results are provided that compare PICPIs with existing interval-based baselines.

stat.ML↗