Deformed Q-learning for High-Dimensional Dynamic Treatment Regimes

Abstract

In high-dimensional dynamic treatment regimes, nonlinear terminal outcome models complicate upstream conditional-mean modeling and inference based on recursively estimated continuation values. We introduce a Deformed Q-learning procedure for general action spaces that accommodates a terminal-stage GLM, random designs, conditional heteroscedasticity, and fine-grained action discretization. The procedure changes the recursive criterion by propagating the terminal deformed optimal Q-function through sparse block-wise linear conditional-mean models. Under the stated models and regularity conditions, we establish consistent and asymptotically normal estimators, accounting for downstream estimation uncertainty in inference for optimal Q-values, deformed optimal Q-values, and their contrasts. We bound population regret relative to the original terminal mean-outcome objective without explicit stage-wise accumulation. Estimation and inference also extend to a growing number of stages under conditions uniform across stages. Simulation studies demonstrate accurate estimation and variable selection. In an empirical application to sepsis management using the MIMIC-III database, the proposed framework supports stage-specific value inference, while an auxiliary action-alignment analysis provides an exploratory signal that early-stage concordance with the learned recommendation may be associated with improved survival.