Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2611.06365

On the Optimality of Implicit Feedback in Markov Decision Processes over Imperfect Links

Romain Planchon·Federico Chiariotti·Federico Mason·Andrea Zanella·Deniz Gunduz
View original source ↗Read source document ↗Export metadata ↓

Abstract

Goal-oriented communication protocols are explicitly designed to support the objectives of target applications, such as the monitoring and control of remote processes, while minimizing data traffic. This study considers a transmitter node that observes the environment state and sends it to a remote agent, which then performs control actions. This problem can be modeled as a remote Partially Observable Markov Decision Process (POMDP), a subclass of decentralized POMDPs. Existing studies limit transmissions through communication costs, but neglect the effect of channel impairments and non-ideal or delayed feedback. In this work, we study the optimal joint transmission and control policies over lossy forward channels without explicit acknowledgments. We introduce \emph{revealing} policies, under which the receiver's action, observed directly by the transmitter, discloses the outcome of the forward transmission, keeping the two agents synchronized without a feedback channel. We derive sufficient conditions for the optimality of time-invariant policies in this implicit feedback regime for general POMDPs, along with strictly weaker sufficient conditions in the special case of contextual bandits. Our simulations over a family of remote POMDPs show that these conditions are conservative, and revealing policies are optimal over a substantially wider parameter range than the conditions warrant.

Explore related subjects

eess.SYcs.SY

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Romain Planchon, Federico Chiariotti, Federico Mason, Andrea Zanella, Deniz Gunduz. 2026-11-04. On the Optimality of Implicit Feedback in Markov Decision Processes over Imperfect Links. https://arxiv.org/abs/2611.06365

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Modeling Rainwater Harvesting Systems with Covered Storage Tank on A Smartphone

A model for simulating rainwater harvesting systems with a covered storage tank is described. The model is simple enough to serve as the simulation engine of a smartphone app for designing such systems. The results of the simulation can be used to find the optimum tank size and the reliability of the system in meeting a given demand. This is illustrated by using the model for designing two such systems, one in an urban environment and the other in an arid rural location. The Python scripts used for the simulations are included as ancillary material.

eess.SY↗
arXiv2026

To Learn or Not to Learn: A Screening Test for Control Problems Before Reinforcement Learning

Reinforcement learning (RL) can be a powerful alternative to classical control methods when standard model-based control is insufficient, e.g., when deriving a suitable model is intractable or impossible. However, in many cases, the choice between model-based and RL-based control is not obvious. Because training RL agents is computationally expensive, RL should be limited to cases where it is expected to outperform model-based control. To the best of our knowledge, there exists no approach to automatically screen out control problems before RL training for which RL is not expected to yield any benefit. In this work, we take the first step towards such an automated screening test. We present a test that identifies problems for which RL-based control is not expected to be superior in two steps. Only control problems that pass both steps should be considered for RL. Part I evaluates the suitability of a provided model for model-based control by analyzing the impact of its uncertainties on the control problem. For this, we use reachset-conformant system identification combined with simulation-based analysis. If the uncertainties are found to be relevant in Part I, Part II then evaluates their learnability using a statistically calibrated dependence analysis. We apply our screening test to several benchmarks, reliably identifying control problems for which RL-based control does not measurably improve upon model-based control, thereby achieving a speed-up of up to $21\times$ compared to trying RL.

eess.SY↗
arXiv2026

State-Feedback Passivity Control for Synchronous Machine Systems

The passivity of a generating station enables network stability to be established from conditions verified locally at each station. For synchronous machines, however, passivity can only be certified under conditions on the operating point, and these conditions fail under heavy loading and weak damping. This paper develops a state-feedback controller that enforces passivity using a storage battery connected at the terminal bus. A bilinear term arising from the product of the rotor-speed and stator-current deviations is treated as a bounded uncertainty, leading to robust positive-real conditions that can be enforced by a state feedback control designed using a single algebraic Riccati equation. The resulting controller guarantees passivity whenever the Riccati equation admits a positive-definite solution, with no separate condition on the operating point. Numerical examples show that existing conditions certify passivity only up to a fraction of the rated current, whereas the proposed state-feedback approach certifies it at the rated operating point of the machine.

eess.SY↗