Beyond endoscopy for the symmetric square representation: The simple trace formula case

Abstract

At the beginning of this century, Langlands introduced a strategy known as \emph{Beyond Endoscopy} to attack the principle of functoriality. Altuğ studied $\mathsf{GL}_2$ over $\mathbb Q$ in the unramified setting for the standard representation. We consider the case with ramification at $S=\{\infty,q_1,\dots,q_r\}$ with $2\in S$ and derive an asymptotic formula for the symmetric square representation adding some additional conditions on the test function so that the trace formula is simple. The limit is nonzero in general and we may detect the dihedral forms by using such limit form of the trace formula which is similar to Venkatesh's thesis. The proof involves a second Poisson summation, computation of the transformed Kloosterman sum and the corresponding series, and giving an asymptotic formula for the main term by residue analysis and using technical analysis to deal with the error term.