Distributed primal-dual algorithm for constrained multi-agent reinforcement learning under coupled policies
This paper investigates constrained multi-agent reinforcement learning (CMARL) in coupled environments, where agents collaboratively maximize the sum of local objectives while satisfying individual safety constraints. Existing studies face two limitations: (1) most rely on independent policies that fail to capture complex interactions in coupled environments; and (2) agents require the global Lagrange multipliers, which are sensitive learned variables whose global sharing risks exposing private agent-specific information. To overcome these issues, we propose a framework where agents adopt coupled policies that depend on both the states and policy parameters of their $κ_p$-hop neighbors, where $κ_p>0$ denotes the coupling distance, and develop a distributed and scalable primal-dual (DSPD) algorithm wherein each agent accesses only information within a prescribed local neighborhood. In the proposed algorithm, agents exchange sensitive parameters only with immediate neighbors over a separate time-varying network, while maintaining local estimates to execute the coupled policy. We establish that the proposed algorithm achieves $ε$-policy stationary convergence with approximation error $\mathcal{O}(γ^{\frac{κ+1}{κ_{p}}})$, where $κ>0$ is the truncated distance and $γ\in(0,1)$ is discount factor. Simulations on a wireless access-control network demonstrate that the proposed algorithm outperforms existing state-of-the-art algorithms, validating its effectiveness.