"Post" Pre-Analysis Plans: Valid Inference for Non-Preregistered Specifications

Reca Sarfati; Vod Vilfort

Home/Archive/arXiv

arXiv · 2510.02507

"Post" Pre-Analysis Plans: Valid Inference for Non-Preregistered Specifications

Reca Sarfati·Vod Vilfort

View original source ↗Read source document ↗Export metadata ↓

Abstract

Pre-analysis plans (PAPs) have become standard in experimental economics research, but it is nevertheless common to see researchers deviating from their PAPs to supplement preregistered estimates with non-prespecified findings. While such ex-post analysis can yield valuable insights, there is broad uncertainty over how to interpret -- or whether to even acknowledge -- non-preregistered results. In this paper, we consider the case of a truth-seeking researcher who, after seeing the data, earnestly wishes to report additional estimates alongside those preregistered in their PAP. We show that, even absent "nefarious" behavior, conventional confidence intervals and point estimators are invalid due to the fact that non-preregistered estimates are only reported in a subset of potential data realizations. We propose inference procedures that account for this conditional reporting. We apply these procedures to Bessone et al. (2021), which studies the economic effects of increased sleep among the urban poor. We demonstrate that, depending on the reason for deviating, the adjustments from our procedures can range from having no difference to an economically significant difference relative to conventional practice. Finally, we consider the robustness of our procedure to certain forms of misspecification, motivating possible heuristic checks and norms for journals to adopt.

Explore related subjects

econ.EM

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Reca Sarfati, Vod Vilfort. 2025-10-02. "Post" Pre-Analysis Plans: Valid Inference for Non-Preregistered Specifications. https://arxiv.org/abs/2510.02507

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Evidence Aggregation for Treatment Choice

Consider a planner who has limited knowledge of the policy's causal impact on a certain local population of interest due to a lack of data, but does have access to the publicized intervention studies performed for similar policies on different populations. How should the planner make use of and aggregate this existing evidence to make her policy decision? Following Manski (2020; Towards Credible Patient-Centered Meta-Analysis, \textit{Epidemiology}), we formulate the planner's problem as a statistical decision problem with a social welfare objective, and solve for an optimal aggregation rule under the minimax-regret criterion. We investigate the analytical properties, computational feasibility, and welfare regret performance of this rule. We apply the minimax regret decision rule to decide whether to enact an active labor market policy based on 14 randomized control trial studies.

econ.EM↗

arXiv2026

Beta-Sorted Portfolios

Beta-sorted portfolios---portfolios comprised of assets with similar covariation with selected risk factors---are a popular tool in empirical finance to analyze models of (conditional) expected returns. Despite their widespread use, little is known of their econometric properties in contrast to comparable procedures such as two-pass regressions. We formally investigate the properties of beta-sorted portfolio returns by casting the procedure as a two-step nonparametric estimator with a nonparametric first step and a beta-adaptive portfolio construction. Our framework rationalizes the well-known estimation algorithm with precise economic and statistical assumptions on the general data-generating process. We provide conditions which ensure valid estimation and inference allowing for a range of hypotheses of interest in financial applications. We show that the rate of convergence of the estimator changes depending on the value of beta. We demonstrate that valid inference depends critically on the object of interest and discuss drawbacks of the widely used Fama-MacBeth variance estimator. To address these limitations, we propose a new variance estimator. We demonstrate the usefulness of our theoretical results in two empirical applications, including one in which we introduce a novel risk factor that captures the business credit cycle and show that it predicts both the cross-sectional and time-series behavior of U.S. stock returns.

econ.EM↗

arXiv2026

Difference-in-Differences with Unpoolable Data

Difference-in-differences (DID) is commonly used to estimate treatment effects but is infeasible in settings where data are unpoolable due to privacy concerns or legal restrictions on data sharing, particularly across jurisdictions. In this study, we identify and relax the assumption of data poolability in DID estimation. We propose an innovative approach to estimate DID with unpoolable data (UN-DID) which can accommodate covariates, multiple groups, and staggered adoption. Through analytical proofs and Monte Carlo simulations, we show that UN-DID and conventional DID estimates of the average treatment effect and standard errors are equal and unbiased in settings without covariates. With covariates, both methods produce estimates that are unbiased, equivalent, and converge to the true value. The estimates differ slightly but the statistical inference and substantive conclusions remain the same. Two empirical examples with real-world data further underscore UN-DID's utility. The UN-DID method allows the estimation of cross-jurisdictional treatment effects with unpoolable data, enabling better counterfactuals to be used and new research questions to be answered.

econ.EM↗