Max-Test-Calibrated Stein Shrinkage with Honest Submodel Selection in Ultra-High-Dimensional Regression

Abstract

Classical preliminary-test and Stein-type estimators interpolate between restricted and full regression fits, but OLS and chi-squared calibration fail when $p\gg n$ and the restriction is data-adaptive. We propose an honest sample-separated framework built around a common selected-null law. Independent selection data extend a mandatory core; separate estimation data pair a complete-design regularized full-model (FM) fit with a fresh exact-null submodel refit. A maximum residual-association statistic assesses all excluded coordinates. Projected Gaussian draws reproduce its conditional null distribution under homoskedastic Gaussian errors, yielding a finite-sample-valid rank test. The inverse moment of the same design-specific law replaces $q-2$ and calibrates preliminary-test (PT), Stein-type (S), and positive-part Stein-type (PS) rules. We derive exact endpoint-risk formulas, honest conditional validity, null-law concentration, inverse-moment consistency, an explicit selection-failure remainder, and endpoint adaptivity without uniform-dominance claims. Gaussian experiments with 2,000 replications and up to 30,000 predictors compare Ridge and LASSO FMs, tuned by frozen-fold cross-validation, under a common submodel, test, and calibration. PS provides large near-null coefficient-risk gains and approaches the relevant full-model risk under strong departures. A separate CPSS-LASSO/CPSS-MCP audit evaluates data-adaptive selection, while a split-sample DepMap study with 19,152 predictors illustrates prediction gains and sensitivity to error assumptions. The HDMaxShrink R package implements the procedure.