Fractional Programming for Kullback-Leibler Divergence in Hypothesis Testing

Abstract

Maximizing the Kullback-Leibler divergence (KLD) is a fundamental problem in waveform design for active sensing and hypothesis testing, as it directly relates to the error exponent of detection probability. However, the associated optimization problem is highly nonconvex due to the intricate coupling of log-determinant and matrix trace terms. Existing solutions often suffer from high computational complexity, typically requiring matrix inversion at every iteration. In this paper, we propose a computationally efficient optimization framework based on fractional programming (FP). Our key idea is to reformulate the KLD maximization problem into a sequence of tractable quadratic subproblems using matrix FP. To further reduce complexity, we introduce a nonhomogeneous relaxation technique that replaces the costly linear system solver with a simple closed-form update, thereby reducing the per-iteration complexity to quadratic order. To compensate for the convergence speed trade-off caused by relaxation, we employ an acceleration method called STEM by interpreting the iterative scheme as a fixed-point mapping. The resulting algorithm achieves significantly faster convergence rates with low per-iteration cost. Numerical results demonstrate that our approach reduces the total runtime by orders of magnitude compared to a state-of-the-art benchmark. Finally, we apply the proposed framework to a multiple random access scenario and a joint integrated sensing and communication scenario, validating the efficacy of our framework in such applications.