Aspiration-based Perturbed Learning Automata in Weakly-Acyclic Games with Noisy Utility Measurements

Abstract

Reinforcement-based learning dynamics may exhibit several limitations when applied in a distributed setup. In (repeatedly-played) multi-player/action strategic-form games, and when each player applies an independent copy of the learning dynamics, convergence to (usually desirable) pure Nash equilibria cannot be guaranteed. Prior work has only focused on a small class of games, namely potential and coordination games. Furthermore, strong convergence guarantees (i.e., almost sure convergence or weak convergence) are mostly restricted to two-player games. To address this main limitation of reinforcement-based learning in repeatedly-played strategic-form games, this paper introduces a novel payoff-based learning scheme for distributed optimization in multi-player/action strategic-form games. We present an extension of perturbed learning automata (PLA), namely aspiration-based perturbed learning automata (APLA), in which each player's probability distribution for selecting actions is reinforced both by repeated selection and an aspiration factor that captures the player's satisfaction level. We provide a stochastic stability analysis of APLA in multi-player positive-utility weakly-acyclic games under the presence of noisy observations. We provide conditions under which convergence is attained (in weak sense) to the set of pure Nash equilibria. A methodology is also derived for calculating the stochastically stable equilibria through the aspiration action functional minimization, which simplifies the derivation of stochastically stable states. Conditions can then be derived for convergence to the Pareto efficient Nash equilibria. To the best of our knowledge, this is the first reinforcement-based learning scheme that provides global convergence guarantees in weakly-acyclic games and in a fully-distributed setup. A Monte-Carlo simulation study validates the derived conclusions.