Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2005.00817

A survey on modern trainable activation functions

Andrea Apicella·Francesco Donnarumma·Francesco Isgrò·Roberto Prevete
View original source ↗Read source document ↗Export metadata ↓

Abstract

In neural networks literature, there is a strong interest in identifying and defining activation functions which can improve neural network performance. In recent years there has been a renovated interest of the scientific community in investigating activation functions which can be trained during the learning process, usually referred to as "trainable", "learnable" or "adaptable" activation functions. They appear to lead to better network performance. Diverse and heterogeneous models of trainable activation function have been proposed in the literature. In this paper, we present a survey of these models. Starting from a discussion on the use of the term "activation function" in literature, we propose a taxonomy of trainable activation functions, highlight common and distinctive proprieties of recent and past models, and discuss main advantages and limitations of this type of approach. We show that many of the proposed approaches are equivalent to adding neuron layers which use fixed (non-trainable) activation functions and some simple local rule that constraints the corresponding weight layers.

Explore related subjects

cs.LGcs.NEstat.ML

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Andrea Apicella, Francesco Donnarumma, Francesco Isgrò, Roberto Prevete. 2021-02-25. A survey on modern trainable activation functions. https://doi.org/10.1016/j.neunet.2021.01.026

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Neural Network Operator-Based Fractal Approximation: Smoothness Preservation and Convergence Analysis

This paper introduces the construction of fractal interpolation functions (FIFs), whose graphs are the attractors of an iterated function system (IFS). Integrating concepts from approximation theory, $α$-fractal functions are constructed, employing shallow neural network operators. Based on the same methodology, we developed fractal interpolation functions using only discrete function values, unlike traditional methods that require each value of the target function. In order to preserve the smoothness of the target function, a method for constructing such FIFs is introduced, employing four-layered neural network operators, i.e., whenever $f \in C^{r}[a,b]$, the corresponding FIF $f^α \in C^{r}[a,b]$. This work uses key approximation theory tools, such as the modulus of continuity and interpolation operators, to develop convergence results and uniform approximation error bounds. To validate the theoretical results obtained, numerical experiments with graphical analysis using Python is provided.

cs.LG↗
arXiv2026

N$^2$: A Unified Python Package and Test Bench for Nearest Neighbor-Based Matrix Completion

Nearest neighbor (NN) methods have re-emerged as competitive tools for matrix completion, offering strong empirical performance and recent theoretical guarantees, including entry-wise error bounds, confidence intervals, and minimax optimality. Despite their simplicity, recent work has shown that NN approaches are robust to a range of missingness patterns and effective across diverse applications. This paper introduces N$^2$, a unified Python package and testbed that consolidates a broad class of NN-based methods through a modular, extensible interface. Built for both researchers and practitioners, N$^2$ supports rapid experimentation and benchmarking. Using this framework, we introduce a new NN variant that achieves state-of-the-art results in several settings. We also release a benchmark suite of real-world datasets, from healthcare and recommender systems to causal inference and LLM evaluation, designed to stress-test matrix completion methods beyond synthetic scenarios. Our experiments demonstrate that while classical methods excel on idealized data, NN-based techniques consistently outperform them in real-world settings.

cs.LG↗
arXiv2026

CUBE: Contrastive Understanding by Balanced Experiments

We introduce CUBE, Contrastive Understanding by Balanced Experiments, to estimate main effects and pairwise Banzhaf interactions over a fixed two-state probe space using shared balanced queries. Each design cancels constant and odd-order contamination from pairwise estimates, while randomization ensures unbiasedness and explicit alias probabilities yield exact mean squared errors without sparsity assumptions or order truncation. Matching risk bounds establish asymptotic minimax optimality within a specified class of nonadaptive, equal-weight, reversal-symmetric balanced contrast estimators, as admissible budgets grow while remaining small relative to the full probe space. Experiments reveal different interaction-specific risks despite identical output distributions and low-order effects, and different sign risks despite equal mean squared errors. Across evaluated tabular models, CUBE generally outperforms several sampling-based estimators, while Faith-Banzhaf-2 achieves the lowest mean error at the largest tested budgets.

cs.LG↗