Solving Zero-Sum Convex Markov Games

Fivos Kalogiannis; Emmanouil-Vasileios Vlatakis-Gkaragkounis; Ian Gemp; Georgios Piliouras

Home/Archive/arXiv

arXiv · 2506.16120

Solving Zero-Sum Convex Markov Games

Fivos Kalogiannis·Emmanouil-Vasileios Vlatakis-Gkaragkounis·Ian Gemp·Georgios Piliouras

View original source ↗Read source document ↗Export metadata ↓

Abstract

We contribute the first provable guarantees of global convergence to Nash equilibria (NE) in two-player zero-sum convex Markov games (cMGs) by using independent policy gradient methods. Convex Markov games, recently defined by Gemp et al. (2024), extend Markov decision processes to multi-agent settings with preferences that are convex over occupancy measures, offering a broad framework for modeling generic strategic interactions. However, even the fundamental min-max case of cMGs presents significant challenges, including inherent nonconvexity, the absence of Bellman consistency, and the complexity of the infinite horizon. We follow a two-step approach. First, leveraging properties of hidden-convex--hidden-concave functions, we show that a simple nonconvex regularization transforms the min-max optimization problem into a nonconvex-proximal Polyak-Lojasiewicz (NC-pPL) objective. Crucially, this regularization can stabilize the iterates of independent policy gradient methods and ultimately lead them to converge to equilibria. Second, building on this reduction, we address the general constrained min-max problems under NC-pPL and two-sided pPL conditions, providing the first global convergence guarantees for stochastic nested and alternating gradient descent-ascent methods, which we believe may be of independent interest.

Explore related subjects

cs.GT cs.LG cs.MA math.OC

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Fivos Kalogiannis, Emmanouil-Vasileios Vlatakis-Gkaragkounis, Ian Gemp, Georgios Piliouras. 2025-06-19. Solving Zero-Sum Convex Markov Games. https://arxiv.org/abs/2506.16120

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Money Burning Mechanism Design: From Welfare to Surplus

We settle the worst-case approximability of consumer-surplus maximization in general multidimensional mechanism-design environments. We do so through two black-box reductions from welfare maximization to the agents' total utility. Our first reduction turns exact welfare maximization into a prior-free, universally truthful and ex-post individually rational mechanism that preserves at least a $1/H_n$ fraction of optimal welfare as expected consumer surplus. The guarantee holds for $n$ agents with arbitrary nonnegative valuations over a finite outcome space, where $H_n$ is the $n$-th harmonic number. The factor $H_n$ is worst-case optimal, including its constant, even for a single-item auction with a known i.i.d. prior and Bayesian incentive compatibility. Our second reduction allows existing truthful welfare approximation mechanisms to be reused for surplus maximization. For valuation classes closed under scaling, it converts any ex-post individually rational, truthful $α$-approximation for welfare with nonnegative payments into an $O(α\log(n))$-approximation for surplus. Our sharp guarantee resolves the welfare-approximation aspect of the open question of Hartline and Roughgarden [2008] on the power of money burning beyond $k$-unit auctions, and the question of Ezra et al. [2025] concerning optimal surplus guarantees for broader valuation classes. It also replaces the outcome-dependent $O(\log|\mathcal{O}|)$ guarantee of Fotakis et al. [2015] with the tight agent-dependent factor $H_n$. These results yield polynomial-time mechanisms with the exact $H_n$ guarantee for gross-substitutes. They also give prior-free, universally truthful approximations of $O(H_n\log^2\log m)$ for XOS valuations and $O(H_n\log^3\log m)$ for subadditive valuations using demand and value queries, where $m$ is the number of items.

cs.GT↗

arXiv2026

Self-Bounding Regret Matching+ in Potential Games and Product-Simplex Optimization

Regret matching+ (RM+) is parameter free, scale invariant, and central to large game solving, but its only general individual-regret guarantee grows as $\sqrt{T}$. A recent ICLR result used this envelope to prove that RM+ reaches an $ε$-stationary point of a smooth objective over a product of simplices in $O(ε^{-4})$ iterations, or $O(ε^{-8})$ from the standard zero initialization. We give an exact one-step conservation law for RM+. It states that forward utility gain pays for both squared state motion and growth of the regret-state norm. Norm growth is at most $\sqrt{m-1}$ times forward gain for $m$ actions, and the coefficient is sharp. This yields four results for unmodified RM+. Its regret on any utility path is controlled by centered temporal variation. Its regret is uniformly bounded under alternating play in every finite exact potential game, resolving an open question and making squared activation gaps summable. Both certified lazy and ordinary cyclic play attain an $ε^{-2}$ exponent. On any smooth, possibly nonconcave simplex objective, RM+ finds an $ε$-KKT point in $O(ε^{-2})$ iterations. Most broadly, for a smooth objective over an arbitrary product of simplices, cyclic block RM+ attains the same $O(ε^{-2})$ exponent from arbitrary initialization, with an explicit trajectory-dependent constant. The proof controls the finite objective loss caused by low-state blocks and then self-bounds every block state and the total squared path length. Complete proofs cover zero states, sharpness, common-profile stationarity, and robust gain dominance. Oracle-normalized diagnostics compare RM+ with predictive and smooth extra-gradient variants on graphical potential games and dense nonconvex objectives.

cs.GT↗

arXiv2026

Randomized Online Fair Division: High-Probability and Expected Realized Fairness

We study randomized algorithms for the fully online allocation of indivisible goods among $n\ge2$ agents with nonnegative additive valuations. Goods arrive sequentially and must be allocated immediately and irrevocably, with only $n$ known in advance. Since exact ex-ante envy freeness and proportionality are readily achievable, while no positive ex-post approximation is possible for the fairness notions considered here, we study the intermediate notions of high-probability fairness and expected realized fairness. Against a non-adaptive adversary, we give a randomized algorithm for proportionality up to one good (PROP1) whose parameter depends only on $n$ and that preserves exact ex-ante envy-freeness and proportionality. At confidence $1-δ$, its PROP1 guarantee improves on independent uniform allocation (Rand) by a factor of $Ω(\log n)$, uniformly over $δ\in(0,1/2]$. As $n\to\infty$, its expected realized PROP1 factor is at least $\frac{3-\sqrt5}{2}-o(1)$. We also show that the expected realized PROP1 factor of Rand is $(1+o(1))/\log n$, yielding an improvement of at least $\bigl(\frac{3-\sqrt5}{2}-o(1)\bigr)\log n$ for our algorithm. For every randomized online algorithm and every positive approximation factor, the success probability can be made arbitrarily small for envy freeness up to any good (EFX) and at most $\frac{n+1}{2n}$ for envy freeness up to one good (EF1). Consequently, every randomized fully online algorithm has an expected realized EFX guarantee of zero and an expected realized EF1 guarantee of at most $\frac{n+1}{2n}$.

cs.GT↗