Adaptive Experimentation Can Hide Model Error:Detection Limits and Falsification Budget

Ramnarayan Choudhary

Home/Archive/arXiv

arXiv · 2611.07529

Adaptive Experimentation Can Hide Model Error:Detection Limits and Falsification Budget

Ramnarayan Choudhary

View original source ↗Read source document ↗Export metadata ↓

Abstract

An experiment can be informative about the parameters of a working model and uninformative about whether that model is wrong. Adaptive design can reinforce this mismatch by repeatedly selecting settings where an incorrect model fits. We bound the power of any valid adequacy test in terms of the experimenter's own sampling distribution, then distinguish information absent from the selected data from information lost by the detector. A computable diagnostic measures unresolved error relative to a declared expansion of the working model. It motivates a training-free rule that reserves part of the design budget for falsification. The empirical record is mixed: the independently specified primary study fails detection eligibility, and a later numerator diagnostic fails its improvement criteria. After the frozen rule fails on fresh continuous-null panels, a disclosed secondary horizon-calibrated revision detects 35.8% of misspecified campaigns versus 31.7% for a class-separation control at T = 240, but has no resolved advantage at T = 480. In a corrected 289-unit measured replay it improves on random design and trails class-discrimination EIG. A conditional futility bound explains why remaining time matters without proving the gate constant or its power. These results support separating parameter learning, model checking and detector performance; they do not establish a universally superior design policy or prospective scientific discovery.

Explore related subjects

stat.ME cs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Ramnarayan Choudhary. 2026-11-05. Adaptive Experimentation Can Hide Model Error:Detection Limits and Falsification Budget. https://arxiv.org/abs/2611.07529

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Bayesian MI-LASSO for Variable Selection on Multiply-Imputed Data

Multiple imputation is widely used for handling missing data in real-world applications. For variable selection on multiply-imputed datasets, however, if selection is performed on each imputed dataset separately, it can result in different sets of selected variables across datasets. MI-LASSO, one of the most commonly used approaches to this problem, regards the same variable across all separate imputed datasets as a group variable and exploits the group LASSO to yield a consistent variable selection across all the multiply-imputed datasets. In this paper, we extend MI-LASSO into the Bayesian framework and propose four Bayesian MI-LASSO models for variable selection on multiply-imputed data, including three shrinkage prior-based and one Spike-and-Slab prior-based methods. To overcome the limitations of traditional, ad hoc threshold-based model selection, we develop a principled four-step projection predictive procedure that provides robust variable selection. This procedure also facilitates approximately calibrated post-selection inference and attains near-nominal coverage under a correct selection model. Simulation studies showed that the Bayesian MI-LASSO outperformed MI-LASSO and other alternative approaches, achieving higher specificity and lower mean squared error across a range of settings. We further demonstrated these methods via a case study using a multiply-imputed dataset from the University of Michigan Dioxin Exposure Study. The R package BMIselect is available on CRAN.

stat.ME↗

arXiv2026

Online conformal inference for multi-step time series forecasting

We study the construction of distribution-free prediction intervals for multi-step time series forecasting. In this setting, forecast errors at different horizons are serially dependent, and feedback is delayed, since the error of an h-step-ahead forecast becomes available only after h periods. We propose autocorrelated multi-step conformal prediction (AcMCP), an online procedure that accommodates delayed feedback and explicitly incorporates the dependence among multi-step forecast errors into conformal calibration. To characterize this dependence, we derive first-order representations of the optimal forecast errors under a general nonstationary autoregressive data-generating process, with explicit bounds on remainder terms. AcMCP guarantees long-run coverage of at least the nominal level at every horizon, without requiring exchangeability or any assumption on how the data distribution evolves over time. We also establish a finite-sample bound on the empirical coverage shortfall, which depends on the forecast horizon and the evaluation sample size. Notably, these guarantees do not rely on the accuracy of the linear approximations. Simulations and real-data applications show that AcMCP adapts to time-varying dynamics and maintains rolling coverage close to the nominal level. By exploiting the dependence among multi-step errors, it often yields narrower intervals than its counterpart without this dependence, with similar coverage, particularly at longer horizons.

stat.ME↗

arXiv2026

Likelihood-free Posterior Density Learning for Uncertainty Quantification in Inference Problems

Generative models and those with computationally intractable likelihoods are widely used to describe complex systems in the natural sciences, social sciences, and engineering. Fitting these models to data requires likelihood-free inference methods that explore the parameter space without explicit likelihood evaluations, relying instead on sequential simulation, which comes at the cost of computational efficiency and extensive tuning. We develop an alternative framework called kernel-adaptive synthetic posterior estimation (KASPE) that uses deep learning to directly reconstruct the mapping between the observed data and a finite-dimensional parametric representation of the posterior distribution, trained on a large number of simulated datasets. We provide theoretical justification for KASPE and a formal connection to the likelihood-based approach of expectation propagation. Simulation experiments demonstrate KASPE's flexibility and performance relative to existing likelihood-free methods including approximate Bayesian computation in challenging inferential settings involving posteriors with heavy tails, multiple local modes, and over the parameters of a nonlinear dynamical system.

stat.ME↗