The coalescent in finite populations with arbitrary, fixed structure

Abstract

The coalescent is a stochastic process representing ancestral lineages in a population undergoing neutral genetic drift. Originally defined for a well-mixed population, the coalescent has been adapted in various ways to accommodate spatial, age, and class structure, along with other features of real-world populations. To further extend the range of population structures to which coalescent theory applies, we formulate a coalescent process for a broad class of neutral drift models with arbitrary -- but fixed -- spatial, age, sex, and class structure, haploid or diploid genetics, and any fixed mating pattern. Here, the coalescent is represented as a random sequence of mappings $\mathcal{C} = \left(C_t\right)_{t=0}^\infty$ from a finite set $G$ to itself. The set $G$ represents the ``sites'' (in individuals, in particular locations and/or classes) at which these alleles can live. The state of the coalescent, $C_t:G \to G$, maps each site $g \in G$ to the site containing $g$'s ancestor, $t$ time-steps into the past. Using this representation, we define and analyze coalescence time, coalescence branch length, mutations prior to coalescence, and stationary probabilities of identity-by-descent and identity-by-state. For low mutation, we provide a recipe for computing identity-by-descent and identity-by-state probabilities via the coalescent. Applying our results to a diploid population with arbitrary sex ratio $r$, we find that measures of genetic dissimilarity, among any set of sites, are scaled by $4r(1-r)$ relative to the even sex ratio case.