Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 1808.06924

General hypergeometric distribution: A basic statistical distribution for the number of overlapped elements in multiple subsets drawn from a finite population

Xing-gang Mao·Xiao-yan Xue
View original source ↗Read source document ↗Export metadata ↓

Abstract

General hypergeometric distribution (GHGD) describes the following distribution: from a finite space containing N elements, select T subsets with each subset contains M[i] (T-1 >= i >= 0) elements, what is the probability that exactly x elements are overlapped exactly t times or at least t times (XLO=t or XLO>=t, T >= t >= 0, here LO is level of overlap)? The classical hypergeometric distribution (HGD) describes the situation of two subsets, while the general situation has not been resolved, despite the overlapped elements has been visualized with the Venn diagram method for about 140 years. GHGD described not only the distribution of XLO=t or XLO>=t that are overlapped in all of the subsets (XLO=T), but also the XLO=t or XLO>=t that are overlapped in a portion of the subsets (LO = t or LO >= t, T >= t >= 0). Here, we developed algorithms to calculate the GHGD and discovered graceful formulas of the essential statistics for the GHGD, including mathematical expectation, variance, and high order moments. In addition, statistical theory to infer a statistically reliable gene set from multiple datasets based on these formulas was established by applying Chebyshev's inequalities.

Explore related subjects

math.STmath.PRstat.MEstat.TH

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Xing-gang Mao, Xiao-yan Xue. 2018-12-12. General hypergeometric distribution: A basic statistical distribution for the number of overlapped elements in multiple subsets drawn from a finite population. https://arxiv.org/abs/1808.06924

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

A note on the distribution of the partial correlation coefficient with nonparametrically estimated marginal regressions

There has been much interest in the nonparametric testing of conditional independence in the econometric and statistical literature, but the simplest and potentially most useful method, based on the sample partial correlation, seems to have been overlooked, its distribution only having been investigated in some simple parametric instances. The present note shows that an easy to apply permutation test based on the sample partial correlation with nonparametrically estimated marginal regressions has good large and small sample properties.

math.ST↗
arXiv2026

Instance-Log-Optimality of Portfolio-Based E-Processes and their Sequential Hypothesis Tests

We consider the problem of sequential hypothesis testing using $e$-processes. For a rich class of composite testing problems---which include bounded mean testing, equal mean testing for bounded random tuples, and some key ingredients of two-sample and independence testing as special cases---we show that any $e$-process satisfying a certain sublinear regret bound is asymptotically and almost surely instance-log-optimal for a composite alternative. This is a strong notion of optimality that has not previously been established for the aforementioned problems, and we provide explicit test supermartingales and $e$-processes satisfying this notion in a more general case. Furthermore, we derive matching lower and upper bounds on the expected rejection time in the high-confidence regime for the resulting sequential tests in all of these cases. The proofs of these results make weak, algorithm-agnostic moment assumptions and rely on a proof technique involving the aforementioned regret and a family of numeraire portfolios. Finally, we discuss how all of these theorems hold in a distribution-uniform sense, a notion of log-optimality that is stronger still and seems to be new to the literature.

math.ST↗
arXiv2026

Common Drivers in Sparsely Interacting Hawkes Processes

We study a multivariate Hawkes process as a model for time-continuous relational event networks. The model does not assume the network to be known, it includes covariates, and it allows for both common drivers, parameters common to all the actors in the network, and also local parameters specific for each actor. We derive rates of convergence for all of the model parameters when both the number of actors and the time horizon tends to infinity. To prevent an exploding network, sparseness is assumed. We also discuss numerical aspects.

math.ST↗