Logical contradictions in the One-way ANOVA and Tukey-Kramer multiple comparisons tests with more than two groups of observations

Abstract

We show that the One-way ANOVA and Tukey-Kramer (TK) tests agree on any sample with two groups. This result is based on a simple identity connecting the Fisher-Snedecor and studentized probabilistic distributions and is proven without any additional assumptions; in particular, the standard ANOVA assumptions (independence, normality, and homoscedasticity (INAH)) are not needed. In contrast, it is known that for a sample with k > 2 groups of observations, even under the INAH assumptions, with the same significance level $α$, the above two tests may give opposite results: (i) ANOVA rejects its null hypothesis $H_0^{A}: μ_1 = \ldots = μ_k$, while the TK one, $H_0^{TK}(i,j): μ_i = μ_j$, is not rejected for any pair $i, j \in \{1, \ldots, k\}$; (ii) the TK test rejects $H_0^{TK}(i,j)$ for a pair $(i, j)$ (with $i \neq j$) while ANOVA does not reject $H_0^{A}$. We construct two large infinite pseudo-random families of samples of both types satisfying INAH: in case (i) for any $k \geq 3$ and in case (ii) for some larger $k$. Furthermore, in case (ii) ANOVA, being restricted to the pair of groups $(i,j)$, may reject equality $μ_i = μ_j$ with the same $α$. This is an obvious contradiction, since $μ_1 = \ldots = μ_k$ implies $μ_i = μ_j$ for all $i, j \in \{1, \ldots, k\}.$ Similar contradictory examples are constructed for the Multivariable Linear Regression (MLR). However, for these constructions it seems difficult to verify the Gauss-Markov assumptions, which are standardly required for MLR.