Information Design in Multi-Agent Reinforcement Learning

Yue Lin; Wenhao Li; Hongyuan Zha; Baoxiang Wang

Home/Archive/arXiv

arXiv · 2305.06807

Information Design in Multi-Agent Reinforcement Learning

Yue Lin·Wenhao Li·Hongyuan Zha·Baoxiang Wang

View original source ↗Read source document ↗Export metadata ↓

Abstract

Reinforcement learning (RL) is inspired by the way human infants and animals learn from the environment. The setting is somewhat idealized because, in actual tasks, other agents in the environment have their own goals and behave adaptively to the ego agent. To thrive in those environments, the agent needs to influence other agents so their actions become more helpful and less harmful. Research in computational economics distills two ways to influence others directly: by providing tangible goods (mechanism design) and by providing information (information design). This work investigates information design problems for a group of RL agents. The main challenges are two-fold. One is the information provided will immediately affect the transition of the agent trajectories, which introduces additional non-stationarity. The other is the information can be ignored, so the sender must provide information that the receiver is willing to respect. We formulate the Markov signaling game, and develop the notions of signaling gradient and the extended obedience constraints that address these challenges. Our algorithm is efficient on various mixed-motive tasks and provides further insights into computational economics. Our code is publicly available at https://github.com/YueLin301/InformationDesignMARL.

Explore related subjects

cs.GT cs.AI cs.LG stat.ML

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Yue Lin, Wenhao Li, Hongyuan Zha, Baoxiang Wang. 2023-10-29. Information Design in Multi-Agent Reinforcement Learning. https://arxiv.org/abs/2305.06807

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Money Burning Mechanism Design: From Welfare to Surplus

We settle the worst-case approximability of consumer-surplus maximization in general multidimensional mechanism-design environments. We do so through two black-box reductions from welfare maximization to the agents' total utility. Our first reduction turns exact welfare maximization into a prior-free, universally truthful and ex-post individually rational mechanism that preserves at least a $1/H_n$ fraction of optimal welfare as expected consumer surplus. The guarantee holds for $n$ agents with arbitrary nonnegative valuations over a finite outcome space, where $H_n$ is the $n$-th harmonic number. The factor $H_n$ is worst-case optimal, including its constant, even for a single-item auction with a known i.i.d. prior and Bayesian incentive compatibility. Our second reduction allows existing truthful welfare approximation mechanisms to be reused for surplus maximization. For valuation classes closed under scaling, it converts any ex-post individually rational, truthful $α$-approximation for welfare with nonnegative payments into an $O(α\log(n))$-approximation for surplus. Our sharp guarantee resolves the welfare-approximation aspect of the open question of Hartline and Roughgarden [2008] on the power of money burning beyond $k$-unit auctions, and the question of Ezra et al. [2025] concerning optimal surplus guarantees for broader valuation classes. It also replaces the outcome-dependent $O(\log|\mathcal{O}|)$ guarantee of Fotakis et al. [2015] with the tight agent-dependent factor $H_n$. These results yield polynomial-time mechanisms with the exact $H_n$ guarantee for gross-substitutes. They also give prior-free, universally truthful approximations of $O(H_n\log^2\log m)$ for XOS valuations and $O(H_n\log^3\log m)$ for subadditive valuations using demand and value queries, where $m$ is the number of items.

cs.GT↗

arXiv2026

Self-Bounding Regret Matching+ in Potential Games and Product-Simplex Optimization

Regret matching+ (RM+) is parameter free, scale invariant, and central to large game solving, but its only general individual-regret guarantee grows as $\sqrt{T}$. A recent ICLR result used this envelope to prove that RM+ reaches an $ε$-stationary point of a smooth objective over a product of simplices in $O(ε^{-4})$ iterations, or $O(ε^{-8})$ from the standard zero initialization. We give an exact one-step conservation law for RM+. It states that forward utility gain pays for both squared state motion and growth of the regret-state norm. Norm growth is at most $\sqrt{m-1}$ times forward gain for $m$ actions, and the coefficient is sharp. This yields four results for unmodified RM+. Its regret on any utility path is controlled by centered temporal variation. Its regret is uniformly bounded under alternating play in every finite exact potential game, resolving an open question and making squared activation gaps summable. Both certified lazy and ordinary cyclic play attain an $ε^{-2}$ exponent. On any smooth, possibly nonconcave simplex objective, RM+ finds an $ε$-KKT point in $O(ε^{-2})$ iterations. Most broadly, for a smooth objective over an arbitrary product of simplices, cyclic block RM+ attains the same $O(ε^{-2})$ exponent from arbitrary initialization, with an explicit trajectory-dependent constant. The proof controls the finite objective loss caused by low-state blocks and then self-bounds every block state and the total squared path length. Complete proofs cover zero states, sharpness, common-profile stationarity, and robust gain dominance. Oracle-normalized diagnostics compare RM+ with predictive and smooth extra-gradient variants on graphical potential games and dense nonconvex objectives.

cs.GT↗

arXiv2026

Randomized Online Fair Division: High-Probability and Expected Realized Fairness

We study randomized algorithms for the fully online allocation of indivisible goods among $n\ge2$ agents with nonnegative additive valuations. Goods arrive sequentially and must be allocated immediately and irrevocably, with only $n$ known in advance. Since exact ex-ante envy freeness and proportionality are readily achievable, while no positive ex-post approximation is possible for the fairness notions considered here, we study the intermediate notions of high-probability fairness and expected realized fairness. Against a non-adaptive adversary, we give a randomized algorithm for proportionality up to one good (PROP1) whose parameter depends only on $n$ and that preserves exact ex-ante envy-freeness and proportionality. At confidence $1-δ$, its PROP1 guarantee improves on independent uniform allocation (Rand) by a factor of $Ω(\log n)$, uniformly over $δ\in(0,1/2]$. As $n\to\infty$, its expected realized PROP1 factor is at least $\frac{3-\sqrt5}{2}-o(1)$. We also show that the expected realized PROP1 factor of Rand is $(1+o(1))/\log n$, yielding an improvement of at least $\bigl(\frac{3-\sqrt5}{2}-o(1)\bigr)\log n$ for our algorithm. For every randomized online algorithm and every positive approximation factor, the success probability can be made arbitrarily small for envy freeness up to any good (EFX) and at most $\frac{n+1}{2n}$ for envy freeness up to one good (EF1). Consequently, every randomized fully online algorithm has an expected realized EFX guarantee of zero and an expected realized EF1 guarantee of at most $\frac{n+1}{2n}$.

cs.GT↗