Stochastic Optimal Control of Decoupled Reflected FBSDEs: A Variational Approach with Residual Reflection Terms

Abstract

This paper studies necessary optimality conditions for stochastic control problems governed by decoupled forward--backward stochastic differential equations with reflection constraints on the backward component. The cost functional is defined through the initial value of a reflected BSDE whose lower obstacle depends on the forward state. The nonsmooth nature of the reflection term prevents the direct application of classical variational methods. Using a penalization approach, spike variations, and duality arguments, we derive a penalized variational relation. A key feature of our method is to isolate the reflection contribution into a residual term, avoiding the differentiation of the nonsmooth penalization operator while preserving a Hamiltonian adjoint structure analogous to the classical stochastic maximum principle. Passing to the limit yields a reflected variational inequality of Pontryagin type, where the obstacle effect is explicitly represented through a residual contribution associated with the reflection term.