Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.39995

How Different are Different Differences-in-Differences? A Large-Scale Meta-Analysis

Peter Hull
View original source ↗Read source document ↗Export metadata ↓

Abstract

I use AI-assisted replication to assess the empirical importance of negative weights in two-way fixed effects (TWFE) estimation, drawing on 1,346 staggered-adoption specifications from 108 economics papers. TWFE estimates are highly correlated with estimates from modern difference-in-differences methods, with observed sign reversals in 10.8-14.5% of comparisons and greater precision for TWFE. Hierarchical models that account for estimation noise imply a 91-98% correlation between estimands and a 0.6-2.1% probability of obtaining a statistically significant TWFE estimate with the opposite sign from the modern estimand. These findings suggest negative weights may be a less serious problem in practice, and illustrate how AI can support large-scale empirical assessments of new econometric methods.

Explore related subjects

econ.EM

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Peter Hull. 2026-10-29. How Different are Different Differences-in-Differences? A Large-Scale Meta-Analysis. https://arxiv.org/abs/2610.39995

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

From Many Models, One: Macroeconomic Forecasting with Reservoir Ensembles

Model combination offers an alternative to selecting a single forecasting model, particularly when individual models are sensitive to random initialization and hyperparameter choices. We study online aggregation through Follow-the-Leader (FTL) and Hedge-type algorithms, with an application to ensembles of Multi-Frequency Echo State Networks (MFESNs) for macroeconomic forecasting. On the theoretical side, we derive Hoeffding- and Bernstein-type expected-regret bounds for FTL and decreasing Hedge under independent and $φ$-mixing expert losses. Under a positive gap between the best expert's expected loss and those of its competitors, the bounds are uniform in the number of forecasting rounds and exhibit logarithmic dependence on the number of experts. Monte Carlo experiments investigate the effects of temporal dependence and the separation between experts' expected losses. Empirically, we construct MFESN ensembles for forecasting U.S. quarterly GDP growth using mixed-frequency macroeconomic and financial indicators. The results show that adaptive aggregation, particularly FTL and AdaHedge, achieves substantial reductions in out-of-sample mean squared forecasting error relative to individual MFESN models and conventional forecast combinations.

econ.EM↗
arXiv2026

Welfare at Risk: Distributionally Robust Bounds for Policy Evaluation

This paper develops a framework to study how the welfare effects of policy interventions are distributed across individuals when those effects are not directly observed. We bound the superquantile of individual welfare changes by the superquantile of conditional average welfare changes, which is typically identified, revealing how gains and losses are spread across the population. We also bound the share of the population whose welfare loss from the policy exceeds any given threshold --- a distribution-free counterpart to average cost--benefit analysis, closer to a poverty-rate measure of policy harm than to a mean welfare effect. We further extend both bounds to allow for ambiguity in the analyst's knowledge of the distribution of observable characteristics: a non-robust bound estimated on one population can be systematically overoptimistic when applied to a different target population, while the robust version, with its ambiguity radius chosen commensurate with the shift, restores a valid guarantee. We illustrate the method in generalized Roy models with subjective participation costs, bounding not only the distribution of welfare changes in the overall population but also, in a result new to this literature, the entire distribution of welfare (and of participation costs) among those who select into treatment --- the treatment-on-the-treated distribution, not merely its mean. We connect these results to the multi-outcome IV framework of \cite{Heckman_Urzua_Vytlacil_2006_REStat}, showing our robust bounds are immune to the instrument-sensitivity problem they document

econ.EM↗
arXiv2026

Inference with Granular Instruments in Large Panels

I develop asymptotic theory for Granular Instrumental Variables (GIV) in large panels with both cross-section ($N$) and sample size ($T$) growing. The granular instrument is constructed from individual shocks that must be estimated from the panel. I show that this estimation has a first-order effect on the asymptotic variance of the GIV estimator. Existing methods do not account for this effect, leading to incorrect standard errors and invalid inference. I derive corrected standard errors that deliver valid inference. I show that instrument strength can change as the cross section grows. This yields strong, nearly-weak, and weak regimes for GIV. In particular, I identify a previously uncharacterized nearly-weak regime in which the instrument vanishes asymptotically, but GIV remains valid. In this regime, if the sample size grows sufficiently fast relative to the cross section, the estimator remains consistent and asymptotically normal at a rate slower than $\sqrt{T}$. Further, I characterize the conditions where GIV exhibits standard weak-instrument behavior and provide Anderson-Rubin confidence intervals. I apply the theory to estimate short-run demand and supply elasticities for copper, natural gas, and crude oil. Across the six elasticities, accounting for instrument estimation changes standard errors by up to a fifth.

econ.EM↗