Universality in randomly connected neural networks: old and new on a conjecture of Geman and Hwang

Abstract

Randomly connected networks are ubiquitous in computer science, physics or biology. A central question is whether, and how, their collective dynamics depend on the choice of the distribution of the random couplings. Forty years ago, Geman and Hwang formulated a precise version of this \emph{universality} question for randomly connected neural networks. We come back to these conjectures, review what was proven since, provide a few new extensions and propose new conjectures in this paper. For $n$ neuron networks with couplings with a mean scaling as $1/n$ and light enough tails, two classes of limits are expected. For \emph{weakly heterogeneous} couplings, with variance $o(1/n)$, the network is predicted to converge to a McKean--Vlasov equation, and a broad literature has established this under general assumptions. For \emph{strongly heterogeneous} couplings, with variance of order $1/n$, the limit is a non-Markovian equation driven by a self-consistent Gaussian process. After landmark results for Gaussian couplings, universality was only recently established for linear interactions, while the continuous-time nonlinear case relevant to neuroscience remained open. Here, in the weakly heterogeneous regime, we prove quenched convergence on every time interval under a fourth-moment assumption only, and convergence in probability under an assumption slightly stronger than a second moment. In the strongly heterogeneous regime, we prove convergence for nonlinear interactions in continuous time, averaged over sub-Gaussian couplings and on every time interval, by a relative-entropy comparison with Gaussian couplings. We then extend the conjecture to heavy-tailed couplings, and propose a limit driven by a self-consistent $α$-stable process, for which we prove existence for every $α\in(0,2)$ and uniqueness for $α\in[1,2)$, and provide extensive numerical evidence in all three cases.