The number of monotone trapezoids with prescribed bottom row

Abstract

We establish an operator formula for the number of monotone trapezoids with prescribed bottom row, generalizing alternating sign matrices. The special case of the formula for monotone triangles previously provided an alternative proof for the enumeration of alternating sign matrices and led to several results on alternating sign triangles and alternating sign trapezoids. The generalization presented in this paper reveals an additional ``hidden operator'' that is annihilated in the special case of monotone triangles, whose discovery was a major challenge. The enumeration formula is conceptually simple: it applies, in addition to the newly discovered hidden operator, an operator ensuring row strictness to the formula for the number of Gelfand--Tsetlin trapezoids. Notably, the top row of the monotone trapezoid is not prescribed. Thus, our result involves a free boundary, which is a novel situation in this area. We also uncover an unexpected relation to the coinvariant algebra and propose a conjecture on its generalization.