The Catastrophic Consequences of Agnosticism for Life Searches and a Possible Workaround

Abstract

Planned and ongoing searches for life, both biological and technological, confront an epistemic barrier concerning false positives - namely, that we don't know what we don't know. The most defensible and agnostic approach is to adopt diffuse (uninformative) priors, not only for the prevalence of life, but also for the prevalence of confounders. We evaluate the resulting Bayes factors between the null and life hypotheses for an idealized experiment with $N_{pos}$ positive labels (biosignature detections) among $N_{tot}$ targets with various priors. Using diffuse priors, the consequences are catastrophic for life detection, requiring at least ${\sim}10^4$ (for some priors ${\sim}10^{13}$) surveyed targets to ever obtain "strong evidence" for life. Accordingly, an HWO-scale survey with $N_{tot}{\sim}25$ would have no prospect of achieving this goal. A previously suggested workaround is to forgo the agnostic confounder prior, by asserting some upper limit on it for example, but we find that the results can be highly sensitive to this choice - as well as difficult to justify. Instead, we suggest a novel solution that retains agnosticism: by dividing the sample into two groups for which the prevalence of life differs, but the confounder rate is global. We show that a $N_{tot}=24$ survey could expect 24% of possible outcomes to produce strong life detections with this strategy, rising to $\geq50$% for $N_{tot}\geq76$. However, AB-testing introduces its own unique challenges to survey design, requiring two groups with differing life prevalence rates (ideally greatly so) but a global confounder rate.