Pattern avoidance in nonnesting permutations

Abstract

Nonnesting permutations are permutations of the multiset $\{1,1,2,2,\dots,n,n\}$ that avoid subsequences of the form $abba$ for any $a\neq b$. These permutations have recently been studied in connection to noncrossing (also called quasi-Stirling) permutations, which are those that avoid subsequences of the form $abab$, and in turn generalize the well-known Stirling permutations. Inspired by the work by Archer et al. on pattern avoidance in noncrossing permutations, we consider the analogous problem in the nonnesting case. We enumerate nonnesting permutations that avoid each set of two or more patterns of length 3, as well as those that avoid some sets of patterns of length 4. We obtain closed formulas and generating functions, some of which involve unexpected appearances of the Catalan and Fibonacci numbers. Our proofs rely on decompositions, recurrences, and bijections.