Feedback Cycles in Exploratory Equilibria

Abstract

Entropy regularization smooths equilibrium policies in time-inconsistent stochastic control. At low temperature, the same Gibbs response can strongly amplify errors in learned rewards and dynamics. We show that the derivative of an exploratory equilibrium is governed by a backward Volterra-parabolic resolvent. Along an aligned positive mode, a lower bound has the same exponential order. A block decomposition identifies the source of the amplification: causal paths contribute powers of 1/tau, whereas a positive feedback cycle can produce exponential growth. At fixed temperature, a local equilibrium branch is twice differentiable with respect to finite-dimensional model parameters, which yields a function-valued delta method. A bounded uniformly elliptic diffusion realizes this path-cycle distinction in every finite dimension. Closing one positive cycle changes the root-n linear-response boundary from a power law to order 1/log n; along the cyclic Perron mode, right-endpoint discretization is relatively consistent exactly when N tau^2 -> infinity. An affine model also gives an exact nonlinear transition at the Lambert-W temperature beta T / W(beta T sqrt(n)). Numerical calculations illustrate these rates.