Optimizing Laguerre-Gaussian Mode Interferometry in Gravitational-Wave Detectors through Modal Degeneracy Breaking

Abstract

Higher-order Laguerre-Gaussian modes can reduce test-mass thermal noise in gravitational-wave interferometers, but their implementation is limited by degeneracy-induced coupling losses in realistic high-finesse cavities. We mitigate these losses by optimizing the mirror curvature to break modal degeneracy. We develop a general framework for designing an axisymmetric corrective test-mass surface profile that lifts the degeneracy of the sixth-order Laguerre-Gaussian subspace while preserving an injected donut-shaped $\mathrm{LG}_{0,6}$ mode. Expanded in a Fourier-Bessel basis, the profile is optimized to maximize the frequency splitting from the nearest unwanted mode while minimizing target-mode loss. For a Virgo-like Fabry-Perot arm cavity, the optimized profile separates the $\mathrm{LG}_{0,6}$ resonance from the nearest same-order resonance, $\mathrm{LG}_{1,4}$, by 2.04% of the free spectral range, reducing unwanted-mode gains below unity at the target-mode resonance and suppressing their resonant enhancement. The profile introduces negligible target-mode loss and improves robustness against astigmatism and realistic mirror surface distortions. Monte Carlo simulations with realistic mirror imperfections yield an average contrast defect of 368 ppm and an average reflected optical loss of 979 ppm, equivalent to a round-trip loss of approximately 3.5 ppm for the modeled cavity. These results substantially improve upon the central anti-reflective mirror mask approach based on selective loss engineering, bringing reflected optical loss within a factor of two of that obtained with the fundamental Gaussian mode under the same simulation conditions. This approach offers a route toward robust donut-shaped Laguerre-Gaussian modes in future precision interferometers with realistic imperfections.