Comprehensive study of the global phase diagram in the triangular $J$-$K$-$Γ$ model

Abstract

The celebrated Kitaev honeycomb model provides an analytically tractable example with an exact quantum spin liquid ground state. While in real materials, other types of interactions besides the Kitaev coupling ($K$) are present, such as the Heisenberg ($J$) and symmetric off-diagonal ($Γ$) terms, and these interactions can also be generalized to a triangular lattice. Here, we carry out a comprehensive study of the $J$-$K$-$Γ$ model on the triangular lattice covering the full parameters region, using the combination of the exact diagonalization, classical Monte Carlo and analytic methods. In the HK limit ($Γ=0$), we find five quantum phases which are quite similar to their classical counterparts. Among them, the stripe-A and dual Néel phase are robust against the $Γ$ term, in particular the stripe-A extends to the region connecting the $K=-1$ and $K=1$ for $Γ<0$. Though the 120$^\circ$ Néel phase also extends to a finite $Γ$, its region has been largely reduced compared to the previous classical result. Interestingly, the ferromagnetic (dubbed as FM-A) phase and the stripe-B phase are unstable in response to an infinitesimal $Γ$ interaction. Moreover, we find five new phases for $Γ\ne 0$ which are elaborated by both the quantum and classical numerical methods. Part of the space previously identified as 120$^\circ$ Néel phase in the classical study is found to give way to the modulated stripe phase. Depending on the sign of the $Γ$, the FM-A phase transits into the FM-B ($Γ>0$) and FM-C ($Γ<0$) phase with different spin orientations, and the stripe-B phase transits into the stripe-C ($Γ>0$) and stripe-A ($Γ<0$). Around the positive $Γ$ point, due to the interplay of the Heisenberg, Kiatev and $Γ$ interactions, we find a possible quantum spin liquid with a continuum in spin excitations.