The case for balanced hypothesis tests and equal-tailed confidence intervals

Abstract

Introduction: there is an ongoing debate about directional inference of two-sided hypothesis tests for which some authors argue that rejecting $θ= θ_0$ does not allow to conclude that $θ> θ_0$ or $θ< θ_0$ but only that $θ\neq θ_0$, while others argue that this is a minor error without practical consequence. Discussion: new elements are brought to the debate. It is shown that the directional interpretation of some non-directional hypothesis tests about Receiver Operating Characteristic (ROC) and survival curves may lead to inflated type III error rates with a probability of concluding that a difference exists in the opposite side of the actual difference that can reach 50% in the worst case. Some of the issues of directional tests also apply to two-sided confidence intervals (CIs). It is shown that equal-tailed CIs should be preferred to shortest CIs. New assessment criteria of two-sided CIs and hypothesis tests are proposed to provide a reliable directional interpretation: partial left-sided and right-sided $α$ error rates for hypothesis tests, probabilities of overestimation and underestimation $α_L$ and $α_U$ and interval half-widths for two-sided CIs. Conclusion: two-sided CIs and two-sided tests are interpreted directionally. This implies that directional interpretation be taken in account in the development and evaluation of confidence intervals and tests.