Antiferromagnetic stripe phase and large-gap insulating ground state of the correlated $\sqrt{3}\times\sqrt{3}$~R30$^{\circ}$-Sn/Si(111) single atomic layer

Abstract

The one-third monolayer Sn layer on Si(111) has long been considered a benchmark system for exploring two-dimensional Mott physics, owing to its narrow bandwidth and sizable on-site Coulomb repulsion. Previous experiments suggested the emergence of a low-temperature Mott insulating phase with an energy gap of only a few tens of meV, while theory predicted a possible antiferromagnetic ordering that remained experimentally elusive. Here, by combining low-temperature scanning tunneling microscopy/spectroscopy with first-principles calculations, we reveal that the $\sqrt{3}\times\sqrt{3}$~R30$^{\circ}$-Sn/Si(111) surface undergoes a transition below 30K into a robust insulating state characterized by a remarkably large gap of about 440 $\pm$ 120 meV at 4K, five to ten times larger than previously reported. Quasiparticle interference imaging uncovers a well-defined $2\sqrt{3}\times\sqrt{3}$~R30$^{\circ}$-Sn/Si(111) superstructure, providing direct evidence for a two-dimensional stripe-like antiferromagnetic order. Ab initio calculations reveal that the silicon substrate stabilizes this phase through strong nonlocal tin-tin interactions, highlighting the decisive role of substrate-driven correlations in the $\sqrt{3}\times\sqrt{3}$~R30$^{\circ}$-Sn/Si(111) system.