Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2603.19290

Neural Dynamics Self-Attention for Spiking Transformers

Dehao Zhang·Fukai Guo·Shuai Wang·Jingya Wang·Jieyuan Zhang·Yimeng Shan·Malu Zhang·Yang Yang·Haizhou Li
View original source ↗Read source document ↗Export metadata ↓

Abstract

Integrating Spiking Neural Networks (SNNs) with Transformer architectures offers a promising pathway to balance energy efficiency and performance, particularly for edge vision applications. However, existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference. Through theoretical analysis, we attribute both limitations to the Spiking Self-Attention (SSA) mechanism: the lack of locality bias and the need to store large attention matrices. Inspired by the localized receptive fields (LRF) and membrane-potential dynamics of biological visual neurons, we propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements. Specifically, we introduce a LRF method into SSA to assign higher weights to neighboring regions, strengthening local modeling and improving performance. Building on this, we approximate the resulting attention computation via charge-fire-reset dynamics, eliminating explicit attention-matrix storage and reducing inference-time memory. Extensive experiments on visual tasks confirm that our method reduces memory overhead while delivering significant performance improvements. These results establish it as a key unit for achieving energy-efficient Spiking Transformers.

Explore related subjects

cs.NEcs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Dehao Zhang, Fukai Guo, Shuai Wang, Jingya Wang, Jieyuan Zhang, Yimeng Shan, Malu Zhang, Yang Yang, Haizhou Li. 2026-03-09. Neural Dynamics Self-Attention for Spiking Transformers. https://arxiv.org/abs/2603.19290

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Persistent Memory Through Triple-Loop Consolidation Under Stochastic Unit Turnover

Dissipative cognitive architectures maintain computation through continuous energy expenditure, where units that exhaust their energy are stochastically replaced with fresh random state. This creates a fundamental challenge: how can persistent, context-specific memory survive when all learnable state is periodically destroyed? Existing memory mechanisms -- including elastic weight consolidation, synaptic intelligence, and surprise-driven gating -- rely on gradient computation and are inapplicable to systems that do not perform it. We introduce Deep Memory (DM), a backpropagation-free persistent memory mechanism operating through a triple-loop consolidation cycle: (1) recording of expert-specific content centroids, (2) seeding of replaced units with stored representations, and (3) stabilization through continuous re-entry. Discrete expert routing via Mixture-of-Experts (MoE) gating is required, in the regimes tested, to prevent the centroid convergence that would render stored memories identical. We derive a Foster-Lyapunov drift bound for the full triple loop, showing that seeding rescales the turnover noise floor. Across $1{,}007$ simulation runs over thirteen blocks: (i) removing stable context-expert binding removes specialization ($\mathrm{MI}=1.10$ vs. $0.001$; $n=91$); (ii) DM achieves $R=0.984$ vs. $0.385$ without memory ($n=16$); (iii) continuous seeding reconstructs representations after interference ($R_\mathrm{recon}=0.978$; one-shot fails; $n=30$); (iv) the mechanism operates within a characterized $(K,p)$ envelope ($n=350$); (v) recording $\times$ seeding is the minimal critical dyad ($n=40$); (vi) associative and reservoir baselines (Hopfield, ESN) are compared under matched turnover ($n=370$). DM is thus a falsifiable, bounded mechanism for persistent memory in backpropagation-free cognitive systems, with functional parallels to hippocampal consolidation.

cs.NE↗
arXiv2026

EvoTreeNAD: Genealogy-Guided Evolution for LLM-Driven Neural Architecture Discovery

AI-driven scientific discovery accelerates research by autonomously developing solutions and designs. Large language model (LLM) agents support this process through iterative generation and evaluation. Yet these iterations alone do not ensure cumulative progress or establish which directions to pursue next. Costly evaluation further constrains the scope of exploration. Neural architecture discovery brings these challenges together, coupling open-ended design with resource-intensive experimentation. We introduce EvoTreeNAD, a genealogy-guided evolutionary algorithm that constructs trainable architectures without a supplied seed or a hand-specified search space. Starting from an empty root, it grows a persistent genealogy in which each new node represents a complete architecture. Top-percentile values computed from each node and its descendants guide lineage selection. Using the selected design history, an Idea Agent proposes a variant and a Code Agent implements it. Each evaluated variant becomes a child node, expanding the genealogy while providing evidence for subsequent lineage selection. Our theoretical analysis establishes the existence of stationary variation regimes as the genealogy grows. Under specified variation assumptions, sustained top-percentile family values quantify the probability of generating high-reward architectures in these regimes. EvoTreeNAD discovers architectures that outperform the compared NAS and NAD baselines, achieving CIFAR-10/100 test errors of $2.05{\pm}0.06\%$ and $15.09{\pm}0.22\%$. On all six MedMNIST-v2 tasks, the discovered architectures surpass the strongest listed baselines. A controlled CIFAR-10 study further shows that EvoTreeNAD outperforms direct generation, best-of-$N$ greedy continuation, and full-family-mean routing.

cs.NE↗
arXiv2026

On Growth and Form, and Function: Reusable Regulatory Handles Control Phenotypic Variation

How phenotypic transformations are implemented by changes in underlying regulatory dynamics remains a central question in developmental biology. Inspired by D'Arcy Thompson's 1917 "On Growth and Form", we ask whether coherent large-scale transformations of morphology can be encoded as low-dimensional modulations of a self-organizing developmental system. We use neural cellular automata (NCAs) as bio-inspired models of distributed development, in which a shared local regulatory network grows target morphologies from a single cell. We apply low-rank adaptation (LoRA) to pretrained NCAs, representing each adapted developmental program as a low-rank modulation of a fixed regulatory scaffold. Horizontal and vertical scaling of a fully grown 2D emoji phenotype can each be implemented by rank-one adaptations. Their linear combinations parametrically control phenotype size, generalize beyond the training distribution, and compose with target-specific adapters. Strikingly, adaptations learned for one phenotype transfer zero-shot across structurally and semantically diverse phenotypes sharing the same reference scaffold, while largely preserving internal features. This suggests reusable system-level hyper-directions of scale rather than morphology-specific transformations. From approximately 25,000 independently trained phenotype-specific NCA adapters with a shared scaffold, we further identify latent low-dimensional directions that functionally control phenotypic variation including scaling, style, and symmetrical fission. Together, our results provide a computational realization of D'Arcy Thompson's remarkable grid transformations in a 2D NCA---a minimal cybernetic tissue in which variations of fully grown emoji phenotypes can be encoded, combined, and controlled through low-dimensional directions in regulatory weight space.

cs.NE↗