arXiv subjects

Matthew Smart

Publications and source records attributed to Matthew Smart.

9 recordsLinked to original sources

arXiv2026

Dynamical principles of habituation across substrates and scales

Habituation is a basic form of learning in which a system's response to repeated stimulation progressively diminishes but eventually recovers when the stimulus is withheld. Long studied in animals, it has increasingly been observed in unicellular organisms and non-living devices such as electronic circuits and neuromorphic materials, suggesting underlying dynamical principles that recur across domains. This review asks what those principles are: given qualitative constraints imposed by habituation on a system's response, what is the minimal dynamical structure that satisfies them? We formalize the classical hallmarks of habituation as behavioral constraints on input--output behavior, show that linear time-invariant systems are structurally incompatible with these constraints, and construct nonlinear motifs---linear fading-memory dynamics composed with static nonlinearities---that exhibit the hallmarks across diverse settings. We relate these motifs to models of specific biological systems and to physical and algorithmic realizations, from analog circuits to transient computation in machine learning.

eess.SY↗

arXiv2026

Attention as In-Context Empirical Bayes: A Two-Stage View via Particle Dynamics

We study minimal attention-only transformers under all-token corruption and show they admit a two-stage empirical Bayes interpretation. A single attention step computes a kernel-weighted posterior mean with respect to the empirical distribution defined by the context. Depth refines this distribution through particle dynamics (Stage 1), while a long-range skip-connection carries the noisy input as a query for posterior inference (Stage 2), revealing distinct statistical roles for depth and attention residuals. The framework isolates a minimal setting in which the context itself induces a depth-dependent energy landscape governing in-context inference. We show that effective denoising can emerge without an explicit noise schedule: a fixed kernel bandwidth and finite integration horizon suffice, yielding a principled depth-noise relationship. We further establish a posterior-mean recovery guarantee for a class of well-behaved priors, where the empirical estimator converges to the Bayes-optimal predictor under asymptotic conditions. Connecting these dynamics to reverse-diffusion limits, our results provide a statistical interpretation of attention as in-context inference via sample-based posterior estimation, without explicit density modeling.

cs.LG↗

arXiv2025

In-context denoising with one-layer transformers: connections between attention and associative memory retrieval

We introduce in-context denoising, a task that refines the connection between attention-based architectures and dense associative memory (DAM) networks, also known as modern Hopfield networks. Using a Bayesian framework, we show theoretically and empirically that certain restricted denoising problems can be solved optimally even by a single-layer transformer. We demonstrate that a trained attention layer processes each denoising prompt by performing a single gradient descent update on a context-aware DAM energy landscape, where context tokens serve as associative memories and the query token acts as an initial state. This one-step update yields better solutions than exact retrieval of either a context token or a spurious local minimum, providing a concrete example of DAM networks extending beyond the standard retrieval paradigm. Overall, this work solidifies the link between associative memory and attention mechanisms first identified by Ramsauer et al., and demonstrates the relevance of associative memory models in the study of in-context learning.

cs.LG↗

arXiv2024

Minimal motifs for habituating systems

Habituation - a phenomenon in which a dynamical system exhibits a diminishing response to repeated stimulations that eventually recovers when the stimulus is withheld - is universally observed in living systems from animals to unicellular organisms. Despite its prevalence, generic mechanisms for this fundamental form of learning remain poorly defined. Drawing inspiration from prior work on systems that respond adaptively to step inputs, we study habituation from a nonlinear dynamics perspective. This approach enables us to formalize classical hallmarks of habituation that have been experimentally identified in diverse organisms and stimulus scenarios. We use this framework to investigate distinct dynamical circuits capable of habituation. In particular, we show that driven linear dynamics of a memory variable with static nonlinearities acting at the input and output can implement numerous hallmarks in a mathematically interpretable manner. This work establishes a foundation for understanding the dynamical substrates of this primitive learning behavior and offers a blueprint for the identification of habituating circuits in biological systems.

nlin.AO↗

arXiv2024

A minimal dynamical system and analog circuit for non-associative learning

Learning in living organisms is typically associated with networks of neurons. The use of large numbers of adjustable units has also been a crucial factor in the continued success of artificial neural networks. In light of the complexity of both living and artificial neural networks, it is surprising to see that very simple organisms -- even unicellular organisms that do not possess a nervous system -- are capable of certain forms of learning. Since in these cases learning may be implemented with much simpler structures than neural networks, it is natural to ask how simple the building blocks required for basic forms of learning may be. The purpose of this study is to discuss the simplest dynamical systems that model a fundamental form of non-associative learning, habituation, and to elucidate technical implementations of such systems, which may be used to implement non-associative learning in neuromorphic computing and related applications.

q-bio.NC↗

arXiv2023

A model of replicating coupled oscillators generates naturally occurring cell networks

When a founder cell and its progeny divide with incomplete cytokinesis, a network forms in which each intercellular bridge corresponds to a past mitotic event. Networks built in this manner are required for gamete production in many animals, and different species have evolved very different final network topologies. While mechanisms regulating network assembly have been identified in particular organisms, we lack a quantitative framework to understand network assembly and inter-species variability. Motivated by cell networks responsible for oocyte production in invertebrates, where the final topology is typically invariant within each species, we devise a mathematical model for generating cell networks: each node is an oscillator, and after a full cycle, the node produces a daughter to which it remains connected. These cell cycle oscillations on the nodes are transient and coupled via diffusion over the network's edges. By variation of three biologically motivated parameters, our model generates nearly all such networks currently reported across invertebrates. Furthermore, small parameter variations can rationalize cases of within-species variation. Because cell networks outside of the ovary often form less deterministically, we propose generalizations of our model to account for different sources of stochasticity.

nlin.AO↗

arXiv2021

Emergent properties of collective gene expression patterns in multicellular systems

Multicellular organisms comprise a diverse collection of stable tissues built from different cell types. It remains unclear how large numbers of interacting cells can precisely coordinate their gene expression during tissue self-organization. We develop a generalized model of multicellular gene expression that includes intracellular and intercellular gene interactions in tissue-like collectives. We show that tuning the intercellular signaling strength results in a cascade of transitions from single-cell autonomy towards different self-organized collective states. Despite an enormous number of possible tissue states, signaling tends to stabilize a small number of compositionally and spatially simple tissue types even for disordered interaction networks. Statistical properties of the stable phenotypes are preserved for different interaction networks and initial conditions. These results provide a theoretical framework to study how collections of cells in distinct organisms robustly self-organize into relatively simple tissues even for complex interaction networks mediated by large numbers of different molecules, and how different stable tissues are maintained in homeostasis and disease. The close alignment between this theoretical model of tissue self-organization and modern sequencing techniques, particularly spatial transcriptomics, will enable future applications in broad biological contexts.

physics.bio-ph↗

arXiv2021

On the mapping between Hopfield networks and Restricted Boltzmann Machines

Hopfield networks (HNs) and Restricted Boltzmann Machines (RBMs) are two important models at the interface of statistical physics, machine learning, and neuroscience. Recently, there has been interest in the relationship between HNs and RBMs, due to their similarity under the statistical mechanics formalism. An exact mapping between HNs and RBMs has been previously noted for the special case of orthogonal (uncorrelated) encoded patterns. We present here an exact mapping in the case of correlated pattern HNs, which are more broadly applicable to existing datasets. Specifically, we show that any HN with $N$ binary variables and $p<N$ arbitrary binary patterns can be transformed into an RBM with $N$ binary visible variables and $p$ gaussian hidden variables. We outline the conditions under which the reverse mapping exists, and conduct experiments on the MNIST dataset which suggest the mapping provides a useful initialization to the RBM weights. We discuss extensions, the potential importance of this correspondence for the training of RBMs, and for understanding the performance of deep architectures which utilize RBMs.

cs.LG↗

arXiv2019

Pleiotropy enables specific and accurate signaling in the presence of ligand cross talk

Living cells sense their environment through the binding of extra-cellular molecular ligands to cell surface receptors. Puzzlingly, vast numbers of signaling pathways exhibit a high degree of cross talk between different signals whereby different ligands act through the same receptor or shared components downstream. It remains unclear how a cell can accurately process information from the environment in such cross-wired pathways. We show that a feature which commonly accompanies cross talk - signaling pleiotropy (the ability of a receptor to produce multiple outputs) - offers a solution to the cross talk problem. In a minimal model we show that a single pleiotropic receptor can simultaneously identify and accurately sense the concentrations of arbitrary unknown ligands present individually or in a mixture. We calculate the fundamental limits of the signaling specificity and accuracy of such signaling schemes. The model serves as an elementary "building block" towards understanding more complex cross-wired receptor-ligand signaling networks.

q-bio.MN↗