Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2103.12499

Initializing ReLU networks in an expressive subspace of weights

Dayal Singh·G J Sreejith
View original source ↗Read source document ↗Export metadata ↓

Abstract

Using a mean-field theory of signal propagation, we analyze the evolution of correlations between two signals propagating forward through a deep ReLU network with correlated weights. Signals become highly correlated in deep ReLU networks with uncorrelated weights. We show that ReLU networks with anti-correlated weights can avoid this fate and have a chaotic phase where the signal correlations saturate below unity. Consistent with this analysis, we find that networks initialized with anti-correlated weights can train faster (in a teacher-student setting) by taking advantage of the increased expressivity in the chaotic phase. Combining this with a previously proposed strategy of using an asymmetric initialization to reduce dead node probability, we propose an initialization scheme that allows faster training and learning than the best-known initializations.

Explore related subjects

cs.LGcond-mat.dis-nncond-mat.stat-mech

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Dayal Singh, G J Sreejith. 2021-05-25. Initializing ReLU networks in an expressive subspace of weights. https://arxiv.org/abs/2103.12499

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Neural Network Operator-Based Fractal Approximation: Smoothness Preservation and Convergence Analysis

This paper introduces the construction of fractal interpolation functions (FIFs), whose graphs are the attractors of an iterated function system (IFS). Integrating concepts from approximation theory, $α$-fractal functions are constructed, employing shallow neural network operators. Based on the same methodology, we developed fractal interpolation functions using only discrete function values, unlike traditional methods that require each value of the target function. In order to preserve the smoothness of the target function, a method for constructing such FIFs is introduced, employing four-layered neural network operators, i.e., whenever $f \in C^{r}[a,b]$, the corresponding FIF $f^α \in C^{r}[a,b]$. This work uses key approximation theory tools, such as the modulus of continuity and interpolation operators, to develop convergence results and uniform approximation error bounds. To validate the theoretical results obtained, numerical experiments with graphical analysis using Python is provided.

cs.LG↗
arXiv2026

N$^2$: A Unified Python Package and Test Bench for Nearest Neighbor-Based Matrix Completion

Nearest neighbor (NN) methods have re-emerged as competitive tools for matrix completion, offering strong empirical performance and recent theoretical guarantees, including entry-wise error bounds, confidence intervals, and minimax optimality. Despite their simplicity, recent work has shown that NN approaches are robust to a range of missingness patterns and effective across diverse applications. This paper introduces N$^2$, a unified Python package and testbed that consolidates a broad class of NN-based methods through a modular, extensible interface. Built for both researchers and practitioners, N$^2$ supports rapid experimentation and benchmarking. Using this framework, we introduce a new NN variant that achieves state-of-the-art results in several settings. We also release a benchmark suite of real-world datasets, from healthcare and recommender systems to causal inference and LLM evaluation, designed to stress-test matrix completion methods beyond synthetic scenarios. Our experiments demonstrate that while classical methods excel on idealized data, NN-based techniques consistently outperform them in real-world settings.

cs.LG↗
arXiv2026

CUBE: Contrastive Understanding by Balanced Experiments

We introduce CUBE, Contrastive Understanding by Balanced Experiments, to estimate main effects and pairwise Banzhaf interactions over a fixed two-state probe space using shared balanced queries. Each design cancels constant and odd-order contamination from pairwise estimates, while randomization ensures unbiasedness and explicit alias probabilities yield exact mean squared errors without sparsity assumptions or order truncation. Matching risk bounds establish asymptotic minimax optimality within a specified class of nonadaptive, equal-weight, reversal-symmetric balanced contrast estimators, as admissible budgets grow while remaining small relative to the full probe space. Experiments reveal different interaction-specific risks despite identical output distributions and low-order effects, and different sign risks despite equal mean squared errors. Across evaluated tabular models, CUBE generally outperforms several sampling-based estimators, while Faith-Banzhaf-2 achieves the lowest mean error at the largest tested budgets.

cs.LG↗