Private & Common Information States in Decentralized Team Equilibrium via Dynamic Programming for POMDPs with Delayed Sharing

Abstract

Witsenhausen, in his seminal 1971 paper [1], introduced decentralized partially observable Markov decision problems (POMDPs), with multiple agents or controls operating under T-step delayed sharing information patterns. A fundamental problem in [1] is the identification of structural properties of optimal strategies that compress the information patterns into multiple information states. In this paper, we develop such structural properties of optimal strategies and associated dynamic programming (DP) equations, using the concept of decentralized sequential team equilibrium (a generalization of person-by-person optimality from static team theory). Within this framework, each strategy is assigned an individual value function conditioned on its delayed sharing information pattern, while the strategies of all other agents are held fixed. The resulting DP framework yields several new DP equations and characterizations of decentralized team equilibrium. Moreover, these DP equations exhibit fundamental properties analogous to those of centralized DP of POMDPs: the optimization in each agent's DP equations is performed over the agent's action space rather than over strategy spaces; each agent's multiple information states satisfy Markov recursions; and a separation principle holds. The DP equations reveal a structural compression property of optimal strategies: each agent compresses its delayed sharing information pattern into three components: 1) a private posterior distribution conditioned on the agent's delayed sharing information pattern, 2) a centralized posterior distribution conditioned on the common information shared by all agents, and 3) the agent's private information component. This structural result substantially extends Witsenhausen's Assertion 8 in [1].