Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.39326

Differential Properties of Objective and Value Functions for Discounted Markov Decision Processes with Finite State and Action Sets

Eugene A. Feinberg·Gaojin He
View original source ↗Read source document ↗Export metadata ↓

Abstract

This paper studies differential properties of objective functions for discrete-time discounted Markov decision processes (MDPs) with finite state and action sets, when differentiation is considered with respect to the discount factor. Given a fixed discount factor for an MDP, we estimate the norms and seminorms of the objective functions, their derivatives, and the value function at a different discount factor. We then use these estimates to prove that objective functions, their derivatives, and value functions are Lipschitz continuous in discount factors. As a result, if an MDP is solved with a fixed discount factor, then we can estimate the value function at a different discount factor without solving it again.\\ %Following these estimations, we also reveal the relations between the seminorms of objective functions, their behavior when the discount factor approaches $1,$ and the discount factors at which the set of optimal policies changes.

Explore related subjects

math.OC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Eugene A. Feinberg, Gaojin He. 2026-10-29. Differential Properties of Objective and Value Functions for Discounted Markov Decision Processes with Finite State and Action Sets. https://arxiv.org/abs/2610.39326

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

A particle system approach towards the global well-posedness of master equations for potential mean field games of control

This paper studies the $N$-particle systems as well as the HJB/master equations for a class of generalized mean field control (MFC) problems and the corresponding potential mean field games of control (MFGC). A local in time classical solution for the HJB equation is generated via a probabilistic approach based on the mean field maximum principle. Given an extension of the so called displacement convexity condition, we obtain the uniform estimates on the HJB equation for the $N$-particle system. Such estimates imply the displacement convexity/semi-concavity and thus the prior estimates on the solution to the HJB equation for generalized MFC problems. The global well-posedness of HJB/master equation for generalized MFC/potential MFGC is then proved thanks to the local well-posedness and the prior estimates. In view of the nature of the displacement convexity condition, such well-posedness is also true for the degenerated case. Our analysis on the $N$-particle system also induces an Lipschitz approximator to the optimal feedback function in generalized MFC/potential MFGC where an algebraic convergence rate is obtained. Furthermore, an alternative approximate Nash equilibrium is proposed based on the $N$-particle system, where the approximation error is quantified thanks to the aforementioned uniform estimates.

math.OC↗
arXiv2026

Stochastic Optimal Control Problems for the Cost-Optimal Management of a Standalone Microgrid

In this paper, we consider a domestic standalone microgrid equipped with local renewable energy generation such as photovoltaic panels, consumption units, and battery storage to balance supply and demand and investigate the stochastic optimal control problem for its cost-optimal management. As a special feature, the manager does not have access to the power grid but has a local generator, making it possible to produce energy using fuel when needed. Such systems are very important for rural electrification, particularly in developing countries. However, these systems are very complex to control due to uncertainties in the weather and environmental conditions, which affect the energy generation and the energy demand. In addition, we assume that the battery and the fuel tank have limited capacities and that the fuel tank can only be filled once at the beginning of the planning period. This leads us to the so-called finite fuel problem. In addition, we allow the energy demand to not always be satisfied, and we impose penalties on unsatisfied demand, the so-called discomfort cost. The main goal is to minimize the expected aggregated cost of generating power using the generator and operating the system. This leads to a mathematical optimization problem. The problem is formulated as a discrete-time stochastic control problem and solved numerically using methods from the theory of Markov decision processes.

math.OC↗
arXiv2026

Penalty-Based Smoothing and Accelerated Inertial Dynamics for Convex Supremum Optimization

We study dual-penalty smoothing for convex functions with a finite-dimensional supremum representation. The regularization yields a differentiable surrogate with controlled approximation error, a single-valued dual maximizer, and explicit gradient formulas. We then study an accelerated inertial dynamic with vanishing damping driven by a time-dependent regularized function whose parameter decreases to zero. The global well-posedness of this system is not an immediate consequence of standard ODE theory, since the gradient of the regularized objective is in general only locally Lipschitz and its local Lipschitz modulus may deteriorate as the smoothing parameter vanishes. We therefore establish existence and uniqueness of global trajectories through a local well-posedness analysis and a continuation argument. Then, assuming that the original problem has a minimizer and under mild integrability conditions on the regularization schedule, we establish an accelerated $\mathcal{O}(t^{-2})$ decay estimate for the regularized and original objective values and, in the regime $α>3$, a sharper $o(t^{-2})$ decay together with weak convergence of trajectories to a minimizer of the original nonsmooth problem via an Opial-type argument. Applications to multiobjective optimization (through Chebyshev/max scalarization) and to distributionally robust optimization (via entropic regularization over ambiguity sets) illustrate the scope of the framework.

math.OC↗