Hamiltonian Primal-Dual Policy Reinforcement Learning for Derivative-Free Model-Based Control

Abstract

Finite-horizon optimal control becomes difficult when a dynamical model can be simulated reliably, but its derivatives are unavailable, discontinuous, or unreliable. Hamiltonian Primal-Dual Policy Optimization (HPDPO) is introduced as a derivative-free, model-based reinforcement learning framework for this setting. The control is represented as a time-indexed finite-horizon policy. Candidate actions are evaluated by deterministic suffix rollouts of the supplied transition model, while a regularized Hamiltonian dual formulation supplies structured future-cost information through block-coordinate multiplier updates. No differentiation of the state dynamics or learned critic is required. The method is validated on nonsmooth problems with exact solutions, an analytical and learned ReLU biomarker cascade, an amyloid Neural ODE trained from longitudinal ADNI data, and three multidimensional-control problems with up to five simultaneous controls. Across these examples, HPDPO closely reproduces exact or reference objective values, trajectories, and control structures. The results show that forward simulation and Hamiltonian dual structure can be combined to solve deterministic finite-horizon control problems involving nonsmooth, learned, or black-box dynamics.