Sharp Generalization for Nonparametric Regression in Interpolation Space by Over-Parameterized Neural Networks Trained with Preconditioned Gradient Descent and Early Stopping

Yingzhen Yang; Ping Li

Home/Archive/arXiv

arXiv · 2407.11353

Sharp Generalization for Nonparametric Regression in Interpolation Space by Over-Parameterized Neural Networks Trained with Preconditioned Gradient Descent and Early Stopping

Yingzhen Yang·Ping Li

View original source ↗Read source document ↗Export metadata ↓

Abstract

We study nonparametric regression using an over-parameterized two-layer neural networks trained with algorithmic guarantees in this paper. We consider the setting where the training features are drawn uniformly from the unit sphere in $\RR^d$, and the target function lies in an interpolation space commonly studied in statistical learning theory. We demonstrate that training the neural network with a novel Preconditioned Gradient Descent (PGD) algorithm, equipped with early stopping, achieves a sharp regression rate of $\cO(n^{-\frac{2αs'}{2αs'+1}})$ when the target function is in the interpolation space $\bth{\cH_K}^{s'}$ with $s' \ge 3$. This rate is even sharper than the currently known nearly-optimal rate of $\cO(n^{-\frac{2αs'}{2αs'+1}})\log^2(1/δ)$~\citep{Li2024-edr-general-domain}, where $n$ is the size of the training data and $δ\in (0,1)$ is a small probability. This rate is also sharper than the standard kernel regression rate of $\cO(n^{-\frac{2α}{2α+1}})$ obtained under the regular Neural Tangent Kernel (NTK) regime when training the neural network with the vanilla gradient descent (GD), where $2α= d/(d-1)$. Our analysis is based on two key technical contributions. First, we present a principled decomposition of the network output at each PGD step into a function in the reproducing kernel Hilbert space (RKHS) of a newly induced integral kernel, and a residual function with small $L^{\infty}$-norm. Second, leveraging this decomposition, we apply local Rademacher complexity theory to tightly control the complexity of the function class comprising all the neural network functions obtained in the PGD iterates. Our results further suggest that PGD enables the neural network to escape the linear NTK regime and achieve improved generalization.

Explore related subjects

stat.ML cs.LG math.ST stat.TH

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Yingzhen Yang, Ping Li. 2025-10-05. Sharp Generalization for Nonparametric Regression in Interpolation Space by Over-Parameterized Neural Networks Trained with Preconditioned Gradient Descent and Early Stopping. https://arxiv.org/abs/2407.11353

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Attack-Resistant Uniform Fairness for Linear and Smooth Contextual Bandits

Modern digital platforms use contextual bandits to allocate valuable exposure and opportunities among competing participants. Fair treatment is therefore an important concern, yet reward maximization alone does not ensure that preferential allocation reflects participants' merits. We develop algorithms for linear and smooth contextual bandits under uniform merit-based fairness, requiring the reward ordering to justify preferential allocation across all contexts and rounds, and study how these guarantees are affected by adversarial reward corruption. Our algorithms achieve $(1-\widetilde O(1/T))$-fairness, with regret that is minimax optimal among fair policies for linear rewards and nearly minimax optimal for smooth rewards. In the linear setting, matching lower bounds identify the price of fairness exactly: minimax regret increases from $\log T$ to $\log^2 T$. For smooth rewards, the cost of fairness is at most polylogarithmic. We further establish a separation between regret and fairness robustness: an $\widetilde O(1)$ corruption budget can cause substantial fairness violations without worsening the regret order. We therefore develop robust algorithms that adapt sampling, estimation, and fairness certification to corruption, which preserve uniform fairness and achieve minimax-optimal and nearly optimal regrets for linear and smooth rewards, respectively. Numerical and semi-synthetic experiments illustrate these findings.

stat.ML↗

arXiv2026

The Cost of Privacy: Rates of Convergence for Parameter Estimation with Differential Privacy

We study the minimax cost of $(\varepsilon,δ)$-differential privacy for mean estimation and Gaussian linear regression in low and high dimensions. For low-dimensional mean estimation, a resampling reduction to fingerprinting yields the privacy contribution $d^2\log(1/δ)/(n^2\varepsilon^2)$ in the stated polynomial-$δ$ regime. For low-dimensional regression, a tracing argument gives the contribution $d^2/(n^2\varepsilon^2)$ under an explicit approximate-DP remainder condition. For sparse mean estimation and sparse regression, a constant-weight packing and a private Fano lemma produce an effective privacy entropy of order $\min\{s\log(ed/s),[\log((e^\varepsilon-1)/δ)]_+\}$ for $δ>0$, up to universal constants and a fixed threshold; for pure DP it is $s\log(ed/s)$. Thus, when $δ$ is polynomially smaller than $\varepsilon$, the pure-DP dependence is retained up to polylogarithmic factors whenever the effective dimension is polylogarithmic in $n$, including regimes with $\varepsilon=o(1)$. Coordinatewise-clipping estimators for means and split-sample noisy-gradient estimators for regression attain the lower bounds up to explicit logarithmic factors. Simulations and data examples illustrate related implementations.

stat.ML↗

arXiv2026

Robust Mixture Models for Algorithmic Fairness Under Latent Heterogeneity

Machine learning models optimized for average performance can perform poorly on vulnerable subpopulations. Existing approaches often rely on groups specified in advance, yet fairness-relevant subgroup structure may be latent, intersectional, and driven by complex interactions among continuous and discrete attributes. We introduce \textbf{ROME} (\textbf{\underline{RO}}bust \textbf{\underline{M}}ixture \textbf{\underline{E}}nsemble), a framework that learns latent group structure while optimizing worst-group predictive performance. ROME connects latent-variable modeling with distributionally robust optimization (DRO) through two complementary approaches: an Expectation-Maximization formulation with robust aggregation for linear models and a neural Mixture-of-Experts formulation for nonlinear settings. Across simulations and three real-world regression datasets, ROME improves worst-group performance while maintaining competitive overall accuracy, including in comparisons with established group-aware and group-label-free robust learning methods. ROME provides a flexible approach to robust prediction when fairness-relevant attributes are available for subgroup discovery but their direct use in group-specific outcome models is restricted.

stat.ML↗