The Yule-$Λ$ Nested Coalescent: Distribution of the Number of Lineages

Abstract

We study a model of a population with individuals sampled from different species. The Yule-$Λ$ nested coalescent describes the genealogy of the sample when each species merges with another randomly chosen species with a constant rate $c$ and the mergers of individuals in each species follow the $Λ$-coalescent. For the Yule-$Λ$ nested coalescent with $c<\int_0^1x^{-1}Λ(dx)<\infty$, where $Λ$ is the measure that characterizes the $Λ$-coalescent, we show that under some initial conditions, the distribution of the number of individual lineages belonging to one species converges weakly to the distribution $ν_c^*$, which is the solution to some recursive distributional equation (RDE) with finite mean. In addition, we show that for some values of $c$, the RDE has another solution with infinite mean.