Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 1909.09502

An Empirical Exploration of Deep Recurrent Connections and Memory Cells Using Neuro-Evolution

Travis J. Desell·AbdElRahman A. ElSaid·Alexander G. Ororbia
View original source ↗Read source document ↗Export metadata ↓

Abstract

Neuro-evolution and neural architecture search algorithms have gained increasing interest due to the challenges involved in designing optimal artificial neural networks (ANNs). While these algorithms have been shown to possess the potential to outperform the best human crafted architectures, a less common use of them is as a tool for analysis of ANN structural components and connectivity structures. In this work, we focus on this particular use-case to develop a rigorous examination and comparison framework for analyzing recurrent neural networks (RNNs) applied to time series prediction using the novel neuro-evolutionary process known as Evolutionary eXploration of Augmenting Memory Models (EXAMM). Specifically, we use our EXAMM-based analysis to investigate the capabilities of recurrent memory cells and the generalization ability afforded by various complex recurrent connectivity patterns that span one or more steps in time, i.e., deep recurrent connections. EXAMM, in this study, was used to train over 10.56 million RNNs in 5,280 repeated experiments with varying components. While many modern, often hand-crafted RNNs rely on complex memory cells (which have internal recurrent connections that only span a single time step) operating under the assumption that these sufficiently latch information and handle long term dependencies, our results show that networks evolved with deep recurrent connections perform significantly better than those without. More importantly, in some cases, the best performing RNNs consisted of only simple neurons and deep time skip connections, without any memory cells. These results strongly suggest that utilizing deep time skip connections in RNNs for time series data prediction not only deserves further, dedicated study, but also demonstrate the potential of neuro-evolution as a means to better study, understand, and train effective RNNs.

Explore related subjects

cs.NE

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Travis J. Desell, AbdElRahman A. ElSaid, Alexander G. Ororbia. 2019-09-27. An Empirical Exploration of Deep Recurrent Connections and Memory Cells Using Neuro-Evolution. https://arxiv.org/abs/1909.09502

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

A Memristive Synapse for Online STDP Learning and Inference in SNNs

This work presents a fully analog memristive synaptic circuit for online spike-timing-dependent plasticity (STDP) learning in spiking neural networks (SNNs). The proposed synapse integrates a local STDP circuit generating gradual timing-dependent conductance updates directly from pre- and post-synaptic spikes. Learning occurs during normal network operation without requiring external digital control or explicit STDP waveform synthesis. Post-layout simulations of the memristive synapse implemented in a 130 nm CMOS technology show spike-timing-dependent conductance adaptation during SNN operation. A 2x2 SNN simulation further illustrates online neuron specialization through unsupervised learning.

cs.NE↗
arXiv2026

Complete Suffix Prediction for Recommendation via Latent Retrieval over Process Graphs

Complete suffix prediction is challenging in sequential decision settings, where the same prefix can remain compatible with several plausible suffixes. We propose a graphbased metric-learning framework that reformulates complete suffix prediction as latent retrieval over process graphs. Prefixes and suffixes are represented as directed attributed graphs and encoded by edge-conditioned graph neural networks, allowing event-level activities and transition-level durations to be modelled jointly. Prefix representations are projected into the latent suffix space through a predictor trained with a joint reconstruction and contrastive objective strengthened using process-aware hard negatives. To stabilise the learned retrieval geometry, spectral normalisation, and retrieval robustness, spectral normalisation is applied to enforce a Lipschitz constraint on both encoders and predictor. Experiments on two real-life process datasets demonstrate that the proposed framework achieves the best overall results across nearly all evaluated criteria. It improves semantic suffix accuracy measured by normalized Damerau-Levenshtein distance, yields strong retrieval quality through Recall@1, Recall@5, and MRR@5, and maintains temporal plausibility according to Mean Absolute Error. These results show that graph-based latent retrieval is an effective alternative to sequential suffix prediction for recommendation-oriented process monitoring under structural and KPI-related constraints.

cs.NE↗
arXiv2026

Event-Native Symbolic-Temporal Spike Encoding Framework for Heterogeneous Cyber Streams

Spiking neural networks (SNNs) have shown promise for sparse, event-driven computation through stateful processing that is naturally compatible with low-power edge hardware. These properties align with cyber monitoring, where data arrives asynchronously, and malicious behavior often emerges through temporal patterns across event sequences. However, cyber streams are not composed solely of continuous numeric signals: their informative structure is also carried by categorical identifiers, irregular timing, and local behavioral context. Traditional rate- and population-based spike encodings are not naturally suited to these heterogeneous semantics, while conventional intrusion detection system (IDS) pipelines typically resolve the mismatch by converting raw events into flows, fixed aggregation windows, or dense tensors. Although useful for conventional classifiers, these transformations introduce buffering latency, obscure native temporal structure, and weaken the computational advantages of event-driven neuromorphic processing. We introduce an event-native symbolic-temporal spike encoding framework that maps heterogeneous cyber events directly into sparse, spike-compatible inputs. By assigning encoding roles to semantic identity, local frequency context, and inter-event timing, the framework preserves categorical semantics and temporal dynamics. We validate the approach on packet-level Network IDS and extend it to message-level CAN IDS, using both domains to evaluate whether the encoding exposes usable structure for recurrent SNNs operating directly on native event streams. Under edge-oriented, $μ$Caspian-aligned hardware constraints, compact recurrent SNNs achieve strong anomaly detection performance, with an operational hybrid metric ($J_{hybrid}$) of 0.987 on Network IDS and 0.980 on CAN IDS.

cs.NE↗