Reinforcement Twinning via Twin-Calibrated Bayesian Policy Search and Replay-Driven Identification

Abstract

The feasibility of updating and synchronising digital twins of actuated systems from operational data depends strongly on how the system is controlled and, consequently, on the data generated during operation. This establishes a direct link between control actuation and model identifiability. This work presents a reinforcement-twinning framework that exploits this link through three computational components. A Gaussian-process surrogate of the physical return combines the twin prediction, used as a calibrated trend, with a discrepancy learned from measured returns, allowing Bayesian policy search to exploit the twin without being tied to its predictions. Candidate policies are proposed around the policy that is optimal for the twin, while measured returns can expand this region. The resulting physical episodes are replayed on the twin after observer-based synchronisation with sparse measurements, and its parameters are identified by combining Bayesian optimisation with gradient-based refinement. Selected information-oriented episodes deploy an excitation policy to improve parameter identifiability. The framework is demonstrated on a Burgers-flow analogue of dynamic-induction control in wind farms, with a deliberately misspecified model structure. Despite substantially biased return predictions, the twin provides a useful starting policy and search region, placing the guided search ahead of unguided Bayesian optimisation from its first physical evaluations. This advantage is maintained when the search region is allowed to expand in response to changes in measured returns, whereas searches over-constrained by the twin-defined region are eventually overtaken by the unguided alternatives. The results support reinforcement twinning as a framework for jointly adapting models and control policies when physical interaction is limited and model discrepancy cannot be neglected.