arXiv subjects

Daniel Adelman

Publications and source records attributed to Daniel Adelman.

4 recordsLinked to original sources

arXiv2026

Thompson Sampling for Infinite-Horizon Discounted Decision Processes

This paper develops a framework for learning in discounted infinite-horizon Markov decision processes (MDPs) with Borel state and action spaces, whose rewards and transitions depend on an unknown parameter.To analyze sampling-based adaptive learning algorithms in this setting, we introduce a canonical probability space that explicitly incorporates sampled parameters into the history of the process. As a performance criterion, we adopt the per-period suboptimality gap used in discounted-MDP regret analysis and specialize it to our parametrized Bayesian setting. Since this quantity captures the remaining loss in future performance from the current period onward, we refer to it as residual regret. We use the expected residual regret to connect discounted-MDP regret analysis with asymptotic discount optimality from adaptive control and the temporal-difference error perspective from reinforcement learning. We then focus on Thompson sampling (TS) in discounted infinite-horizon MDPs. Under assumptions that extend those used in prior work on finite state and action spaces to the Borel setting, we show that the expected residual regret for TS converges to zero exponentially fast. We further show that, under mild conditions ensuring the existence of the relevant limits, the ample-path residual regret converges to zero almost surely and TS achieves complete learning.

stat.ML↗

arXiv2026

Optimizing Treatment Allocation to Maximize the Health of a Population

Recent shifts in global health priorities have positioned Population Health Management (PHM) as a central area of focus. However, optimizing PHM strategies presents several challenges: managing high-dimensional patient covariates, tracking their evolution and long-term response to interventions, and accounting for the inflow and outflow of individuals within the population. In this paper, we propose a novel approach based on Measurized MDPs that integrates these components. We consider a setting in which a treatment with population-level benefits is available but scarce, and model an MDP that optimizes the long-term distribution of the healthcare population under expected capacity constraints. This formulation allows us to bypass both the dimensionality and practical challenges of handling and tracking individual patient covariates across the population. To ensure ethical compliance, we introduce a non-maleficence constraint that limits the allowable mortality rate. To solve the resulting infinite-dimensional problem, we use ADP and reduce the task to identifying a finite set of high-performing treated and untreated patients. Despite the complexity of the underlying structure, our approach yields a simple, clinically implementable index policy: a patient is selected for treatment if their adjusted impactability exceeds a specified threshold. The adjusted impactability captures the long-term consequences of receiving or not receiving treatment. While straightforward to apply, the policy remains flexible and can incorporate general machine learning models. Using CMS data, we show that our policy yields a statistically significant improvement over a myopic benchmark. This advantage increases with the time horizon, consistent with the forward-looking nature of our policy. At the longest horizon tested, this corresponds to over 1,500 additional home days annually per 1,000 patients.

math.OC↗

arXiv2026

Measurized Markov Decision Processes

In this paper, we explore lifting Markov Decision Processes (MDPs) to the space of probability measures and consider the so-called measurized MDPs: deterministic processes where states are probability measures on the original state space, and actions are stochastic kernels on the original action space. We show that measurized MDPs are a generalization of stochastic MDPs, thus the measurized framework can be deployed without loss of fidelity. Bertsekas and Shreve studied similar deterministic MDPs under the discounted infinite-horizon criterion in the context of universally measurable policies. Here, we also consider the long-run average reward case, but we cast lifted MDPs within the semicontinuous-semicompact framework of Hernández-Lerma and Lasserre. This makes the lifted framework more accessible as it entails (i) optimal Borel-measurable value functions and policies, (ii) reasonably mild assumptions that are easier to verify than those in the universally-measurable framework, and (iii) simpler proofs. In addition, we showcase the untapped potential of lifted MDPs by demonstrating how the measurized framework enables the incorporation of constraints and value function approximations that are not available from the standard MDP setting. Furthermore, we introduce a novel algebraic lifting procedure for any MDP, showing that non-deterministic measure-valued MDPs can emerge from lifting MDPs impacted by external random shocks.

math.OC↗

arXiv2025

Dynamic Basis Function Generation for Network Revenue Management

This paper introduces an algorithm that dynamically generates basis functions to approximate the value function in Network Revenue Management. Unlike existing algorithms sampling the parameters of new basis functions, this Nonlinear Incremental Algorithm (NLIAlg) iteratively refines the value function approximation by optimizing these parameters. For larger instances, the Two-Phase Incremental Algorithm (2PIAlg) modifies NLIAlg to leverage the efficiency of LP solvers. It reduces the size of a large-dimensional nonlinear problem and transforms it into an LP by fixing the basis function parameters, which are then optimized in a second phase using the flow imbalance ideas from Adelman and Klabjan (2012). This marks the first application of these techniques in a stochastic setting. The algorithms can operate in two modes: (1) Standalone mode, to construct a value function approximation from scratch, and (2) Add-on mode, to refine an existing approximation. Our numerical experiments indicate that while NLIAlg and 2PIAlg in standalone mode are only feasible for small-scale problems, the heuristic version of 2PIAlg (H-2PIAlg) in add-on mode, using the Affine Approximation and exponential ridge basis functions, can handle extremely large instances that may cause benchmark network revenue management methods to run out of memory. In these scenarios, H-2PIAlg delivers substantially better policies and upper bounds than the Affine Approximation. Furthermore, H-2PIAlg achieves higher average revenues in policy simulations compared to network revenue management benchmarks in instances with limited capacity.

math.OC↗