Data-Driven Robust Reinforcement Learning Control of Uncertain Nonlinear Systems: Towards a Fully-Automated, Insulin-Based Artificial Pancreas

Abstract

In this paper, a novel robust tracking control scheme for a general class of discrete-time nonlinear systems affected by unknown bounded uncertainty is presented. By solving a parameterized optimal tracking control problem subject to the unknown nominal system and a suitable cost function, the resulting optimal tracking control policy can ensure closed-loop stability by achieving a sufficiently small tracking error for the original uncertain nonlinear system. The computation of the optimal tracking controller is accomplished through the derivation of a novel Q-function-based $λ$-Policy Iteration algorithm. The proposed algorithm not only enjoys rigorous theoretical guarantees, but also avoids technical weaknesses of conventional reinforcement learning methods. By employing a data-driven, critic-only least squares implementation, the performance of the proposed algorithm is evaluated to the problem of fully-automated, insulin-based, closed-loop glucose control for patients diagnosed with Type 1 and Type 2 Diabetes Mellitus. The U.S. FDA-accepted DMMS.R simulator from the Epsilon Group is used to conduct a comprehensive in silico clinical campaign on a rich set of virtual subjects under completely unannounced meal and exercise settings. Simulation results underline the superior glycaemic behavior achieved by the derived approach, as well as its overall maturity for the design of highly-effective, closed-loop drug delivery systems for personalized medicine.