Associative Memory for Non-Stationary Environments: A Self-Sizing Generalization of Hopfield Networks

Abstract

The Hopfield network made associative memory (AM) the model system of neural computation, but it solves the problem only for a \emph{stationary} world: a fixed set of memories, stored once into frozen weights. Real environments are non-stationary (e.g., memories arrive over time, drift, recur, and must be told apart from noise), where the classical formulation fails by catastrophic interference (the palimpsest problem) and by a capacity fixed in advance. We argue that this is not a peripheral limitation but the crux: under non-stationarity, memory and learning cease to be separate problems, and \emph{adaptation}, rather than one-shot optimization, becomes the operative capacity. We give a fresh formulation of the AM problem for non-stationary environments and a \emph{self-sizing} continual associative memory that generalizes Hopfield's: it stores new memories without erasing old ones (no forgetting), re-binds drifting and recurring memories, allocates a genuinely new memory only for true novelty, and grows its store to the environment's \emph{intrinsic} memory demand and no further. We rigorously show that this demand is the Urysohn width of the problem and can be estimated from data via a contrastive-similarity (CS) operator. The memory's size converges to this capacity online with no preset value and no validation search, matching an oracle capacity search. We use experiments with synthetic datasets to show that the generalization buys self-sizing and retention under non-stationarity, \emph{not} higher per-item recall fidelity, on which it matches strong baselines.