Reinforcement Learning, Optimal Control, and Bayesian Filtering in Data Assimilation

Abstract

We give a finite-horizon variational formulation that places Bayesian filtering and smoothing, variational data assimilation, KL-regularized control, and Kalman-type methods inside one mathematically explicit hierarchy. For a discrete-time hidden Markov model and any admissible one-step candidate law $q_t$, We prove $J_t(q_t)=\mathbb{E}_{q_t}\!\left[-\log p(y_t\mid X_t)\right] +\mathrm{KL}\!\left(q_t\|p_t^f\right) =\mathrm{KL}\!\left(q_t\|p_t^a\right)-\log p(y_t\mid y_{0:t-1})$, and, for any admissible path law $q$, $J_{\mathrm{path}}(q)=\mathbb{E}_{q}\!\left[-\sum_{t=0}^{T}\log p(y_t\mid X_t)\right] +\mathrm{KL}\!\left(q\|p(x_{0:T})\right) =\mathrm{KL}\!\left(q\|p(x_{0:T}\mid y_{0:T})\right)-\log p(y_{0:T})$. These identities determine the evidence as the global infimum and make the analysis and smoothing posteriors the unique minimizers whenever those posterior laws belong to the admissible classes. This separates targets that are often conflated: strong- and weak-constraint 4D-Var are MAP estimators under the stated Gaussian assumptions; KL-regularized control recovers the Bayesian posterior only when the passive dynamics, likelihood cost, temperature, and a restrictive representability condition on the policy class are all matched correctly; and the linear-Gaussian specialization yields the Kalman analysis exactly. The ensemble Kalman filter then appears as a Gaussian and finite-ensemble approximation to the forecast-to-analysis map, exact only in the linear-Gaussian infinite-ensemble limit. This framework also clarifies RMSE-based RL data assimilation: such rewards may define effective estimators or pseudo-posteriors, but not exact posterior recovery unless they realize the likelihood-plus-KL objective.