Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2407.06887

Risk-averse optimization of total rewards in Markovian models using deviation measures

Christel Baier·Jakob Piribauer·Maximilian Starke
View original source ↗Read source document ↗Export metadata ↓

Abstract

This paper addresses objectives tailored to the risk-averse optimization of accumulated rewards in Markov decision processes (MDPs). The studied objectives require maximizing the expected value of the accumulated rewards minus a penalty factor times a deviation measure of the resulting distribution of rewards. Using the variance in this penalty mechanism leads to the variance-penalized expectation (VPE) for which it is known that optimal schedulers have to minimize future expected rewards when a high amount of rewards has been accumulated. This behavior is undesirable as risk-averse behavior should keep the probability of particularly low outcomes low, but not discourage the accumulation of additional rewards on already good executions. The paper investigates the semi-variance, which only takes outcomes below the expected value into account, the mean absolute deviation (MAD), and the semi-MAD as alternative deviation measures. Furthermore, a penalty mechanism that penalizes outcomes below a fixed threshold is studied. For all of these objectives, the properties of optimal schedulers are specified and in particular the question whether these objectives overcome the problem observed for the VPE is answered. Further, the resulting algorithmic problems on MDPs and Markov chains are investigated.

Explore related subjects

cs.LO

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Christel Baier, Jakob Piribauer, Maximilian Starke. 2024-07-09. Risk-averse optimization of total rewards in Markovian models using deviation measures. https://arxiv.org/abs/2407.06887

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Arithmetics within the Linear Time Hierarchy

We identify fragments of the arithmetic $S_1$ that enjoy nice closure properties and have exact characterization of their definable multifunctions. To do this, in the language of $S_1$, $L_1$, starting from the formula classes, $Σ^{\mathsf b}_{i}$, which ignore sharply bounded quantifiers when determining quantifier alternations, we define new syntactic classes by counting bounded existential sharply bounded universal quantifiers blocks. Using these, we define arithmetics: $\breve{S}^{i}_{1}$, $TLS^i_1$ and $TSC^i_1$. $\breve{S}^{i}_{1}$ consists of open axioms for the language symbols and length induction for one of our new classes, $SIUT_{i,1}^{\{p(|id|)\}}$. $TLS^i_1$ and $TSC^i_1$ are defined using axioms related to dependent choice sequences for formulas from two other classes within $Σ^{\mathsf b}_{i}$. We prove for $i \geq 1$ that $$TLS^i_1 \subseteq TSC^i_1 \subseteq \breve{S}^{i}_{1} \preceq_{\forall B(SITT_{i+1}^{\{p(|id|)\}})} TLS^{i+1}_1$$ and that the $SITT_{i}^{\{p(|id|)\}}$-definable in $TLS^i_1$ (resp. $SITT_{i}^{\{2^{p(||id||)}\}}$-definable in $TSC^i_1$) multifunctions are $L_1$-$FLOGSPACE^{SIT_{i,1}}[wit]$ (resp. $L_1$-$FSC^{SIT_{i,1}}[wit]$). These multifunction classes are respectively the logspace or $SC$ (poly-time, polylog-space) computable multifunctions whose output is bound by a term in $L_1$ and that have access to a witness oracle for another restriction on the $Σ^{\mathsf b}_{i}$ formulas, $SIT_{i,1}$. For the $i=1$ cases, this simplifies respectively to the functions in logspace and $SC$, Steve's Class, poly-time, polylog-space. We prove independence results related to the Matiyasevich Robinson Davis Putnam Theorem (MRDP) and to whether our theories prove simultaneous nondeterministic polynomial time, sublinear space is equal to co-nondeterministic polynomial time, sublinear space.

cs.LO↗
arXiv2026

TREBL -- A Relative Complete Temporal Event-B Logic. Part I: Theory

The verification of liveness conditions is an important aspect of state-based rigorous methods. This article addresses the extension of the logic of Event-B to a powerful logic, in which properties of traces of an Event-B machine can be expressed. However, all formulae of this logic are still interpreted over states of an Event-B machine rather than traces. The logic exploits that for an Event-B machine $M$ a state $S$ determines all traces of $M$ starting in $S$. We identify a fragment called TREBL of this logic, in which all liveness conditions of interest can be expressed, and define a set of sound derivation rules for the fragment. We further show relative completeness of these derivation rules in the sense that for every valid entailment of a formula $φ$ one can find a derivation, provided the machine $M$ is sufficiently refined. The decisive property is that certain variant terms must be definable in the refined machine. We show that such refinements always exist. Throughout the article several examples from the field of security are used to illustrate the theory.

cs.LO↗
arXiv2026

Scenes: A Meta-Logical Algebra for Mutable State

Modelling of mutable state spaces and precisely describing how variables are manipulated in a program is a fundamental problem in compositional verification. Though we can make use of the embedded abstract syntax of a program for such analysis, this runs contrary to the shallow-embedding approach, and hampers efficient proof automation. On the other hand, lenses and prisms provide an elegant algebraic foundation for modelling state, which provide sufficient structure to provide meta-logical program analysis, but without requiring a deep embedding. Nevertheless lenses, as complex algebraic objects, cannot easily be combined, complemented, or collected in sets. In this paper we contribute an accompanying algebraic structure called the scene, which allows us to characterise the set of variables, or coordinates, in a state space. Scenes intuitively correspond to sets of lenses, but like lenses they are purely semantic algebraic objects. We demonstrate that scenes provide us with sufficient structure to characterise the lens-based meta-logical properties, like independence and equivalence. Moreover, we introduce the notion of a scene space, analogous to a vector space, which allows us to recover a set-like algebraic structure. Finally, we show how scenes allow us to characterise the free and bound variables of expressions and programs, without any need for syntax, and demonstrate their use for reasoning about programs by deriving reasoning principles for the parallel composition operator.

cs.LO↗