Reinforcement learning for constrained stochastic linear--quadratic control in continuous time

Abstract

We study continuous-time stochastic linear--quadratic control with a scalar state and nonnegative control constraints. With quadratic running costs, randomized exploration can change the optimal mean feedback. We introduce an auxiliary problem without entropy in which the conditional policy mean is nonnegative and the covariance is prescribed in proportion to the squared state. Moment reduction and state separation yield two regional Riccati equations and characterize the finite-horizon value function and optimal conditional mean. We give a sufficient control--cost compatibility condition under which this mean coincides with the optimal feedback of the original constrained problem. For time-invariant coefficients, we characterize the undiscounted infinite-horizon optimum and establish mean square asymptotic stability under the stated assumptions. For unknown dynamics, we develop model-free actor--critic schemes for finite and infinite horizons. Numerical experiments compare the learned policies with their analytical targets and examine the effect of exploration on the optimal mean.