Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Sai Niranjan Ramachandran; Suvrit Sra

Home/Archive/arXiv

arXiv · 2609.02373

Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

Sai Niranjan Ramachandran·Suvrit Sra

View original source ↗Read source document ↗Export metadata ↓

Abstract

We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which architectural symmetries force subnetworks to merge in discrete simultaneous blocks rather than one at a time. These structural transitions register as variance spikes in a macroscopic order parameter, echoing physical phase transitions. We further show this trapping mechanism and its associated scaling cascade extend to Adam and AdamW under an explicit heavy-tailed noise model.

Explore related subjects

cs.LG cond-mat.dis-nn cond-mat.stat-mech cs.AI

Keep this discovery

BibTeX RIS

Sai Niranjan Ramachandran, Suvrit Sra. 2026-09-02. Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance. https://arxiv.org/abs/2609.02373

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cond-mat.dis-nn subject cond-mat.stat-mech subject cs.AI subject cs.LG subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2026

On the Existence of Consistent Adversarial Attacks in High-Dimensional Linear Classification

What fundamentally distinguishes an adversarial attack from a misclassification due to limited model expressivity or finite data? In this work, we investigate this question in the setting of high-dimensional binary classification, where statistical effects due to limited data availability play a central role. We introduce a new error metric that precisely capture this distinction, quantifying model vulnerability to consistent adversarial attacks -- perturbations that preserve the ground-truth labels. Our main technical contribution is an exact and rigorous asymptotic characterization of these metrics in both well-specified models and latent space models, revealing different vulnerability patterns compared to standard robust error measures. The theoretical results demonstrate that as models become more overparameterized, their vulnerability to label-preserving perturbations grows, offering theoretical insight into the mechanisms underlying model sensitivity to adversarial attacks.

stat.ML↗

arXiv2026

Shortcomings and capacities of real-constrained neural networks in complex spaces

We find the asymptotic ratio between the storage capacities when enforcing real pre-activations in a complex hypothesis class as opposed to complex ones in the same class. We use weights drawn from the complex Gaussian, which converge asymptotically in norm to the square root of dimension almost surely. Our methods depend on Gardner volume-type comparisons at critical capacity. Our proof relies on an application of the Harish-Chandra-Itzykson-Zuber (HCIZ) formula, nonstandard in literature. With the HCIZ formula, we may obtain a more robust approximation for the final asymptotic ratio. This strategy is applicable to our work specifically since we integrate over the unitary and orthogonal compact manifolds, facilitated via the Weyl integration formula and the Haar measure.

cs.LG↗

arXiv2026

Quantum Circuit and Tensor Network Implementation of the 2D Acoustic Wave Equation

We present a cohesive framework for simulating seismic wave propagation utilizing quantum computing paradigms and their classical tensor network equivalents. We detail a quantum circuit-based formulation for the explicit finite-difference time-domain (FDTD) solution of the two-dimensional acoustic wave equation and map this quantum architecture onto a tensor train representation, namely for Matrix Product State (MPS). The MPS solver enables deterministic simulation of large-scale wavefield dynamics on classical high-performance computing systems. We demonstrate the MPS representation by computing 2D seismic wavefields on the Marmousi model. Our results indicate that the MPS representation is a viable direction for computing and scaling wavefield propagation.

quant-ph↗