Scalable Analysis of Bipartite Experiments

Liang Shi; Edvard Bakhitov; Kenneth Hung; Brian Karrer; Charlie Walker; Monica Bhole; Okke Schrijvers

Home/Archive/arXiv

arXiv · 2402.11070

Scalable Analysis of Bipartite Experiments

Liang Shi·Edvard Bakhitov·Kenneth Hung·Brian Karrer·Charlie Walker·Monica Bhole·Okke Schrijvers

View original source ↗Read source document ↗Export metadata ↓

Abstract

Bipartite Experiments are randomized experiments where the treatment is applied to a set of units (randomization units) that is different from the units of analysis, and randomization units and analysis units are connected through a bipartite graph. The scale of experimentation at large online platforms necessitates both accurate inference in the presence of a large bipartite interference graph, as well as a highly scalable implementation. In this paper, we describe new methods for inference that enable practical, scalable analysis of bipartite experiments: (1) We propose CA-ERL, a covariate-adjusted variant of the exposure-reweighted-linear (ERL) estimator [9], which empirically yields 60-90% variance reduction. (2) We introduce a randomization-based method for inference and prove asymptotic validity of a Wald-type confidence interval under graph sparsity assumptions. (3) We present a linear-time algorithm for randomization inference of the CA-ERL estimator, which can be easily implemented in query engines like Presto or Spark. We evaluate our methods both on a real experiment at Meta that randomized treatment on Facebook Groups and analyzed user-level metrics, as well as simulations on synthetic data. The real-world data shows that our CA-ERL estimator reduces the confidence interval (CI) width by 60-90% (compared to ERL) in a practical setting. The simulations using synthetic data show that our randomization inference procedure achieves correct coverage across instances, while the ERL estimator has incorrectly small CI widths for instances with large true effect sizes and is overly conservative when the bipartite graph is dense.

Explore related subjects

stat.ME

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Liang Shi, Edvard Bakhitov, Kenneth Hung, Brian Karrer, Charlie Walker, Monica Bhole, Okke Schrijvers. 2025-11-04. Scalable Analysis of Bipartite Experiments. https://arxiv.org/abs/2402.11070

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Switchback Experiments under Geometric Mixing

The switchback is an experimental design that measures treatment effects by repeatedly turning an intervention on and off for a whole system. Switchback experiments are a robust way to overcome cross-unit spillover effects; however, they are vulnerable to bias from temporal carryovers. In this paper, we consider properties of switchback experiments in Markovian systems that mix at a geometric rate. We find that, in this setting, standard switchback designs suffer considerably from carryover bias: Their estimation error decays as $T^{-1/3}$ in terms of the experiment horizon $T$, whereas in the absence of carryovers a faster rate of $T^{-1/2}$ would have been possible. We also show, however, that judicious use of burn-in periods can considerably improve the situation, and enables errors that decay as $\log(T)^{1/2}T^{-1/2}$. Our formal results are mirrored in an empirical evaluation.

stat.ME↗

arXiv2026

Saddlepoint approximations for plug-in resampling

Resampling-based procedures can improve on normal approximations in sparse, large-scale testing problems, but their computational cost can be prohibitive. We recognize that several existing procedures belong to a faster plug-in resampling subclass, fixing fitted nuisance parameters during resampling. When the resampled statistic is a sum of conditionally independent terms, the saddlepoint approximation (SPA) for the resampling $p$-value offers further acceleration, replacing resampling with an analytical tail approximation. However, standard Edgeworth-based approximation-error bounds impose regularity conditions that are hard to verify for plug-in resampling laws. We use an alternative approach to establish a finite-sample relative-error bound for the Lugannani-Rice approximation under more tractable conditions, which we apply in two contexts. In statistical genetics, we identify response resampling procedures as the targets of existing SPAs and establish guarantees in a representative setting. In conditional independence testing, we introduce spaCRT, an SPA for the distilled conditional randomization test (dCRT), which has been applied successfully in biology. Our rates quantify the effects of sparsity and signal strength, with matching lower bounds in special cases. We additionally establish asymptotic Type-I error control of the corresponding plug-in resampling procedures under growing sparsity. In simulations and single-cell CRISPR data analysis, spaCRT closely approximates dCRT $p$-values and preserves its statistical performance while accelerating computation by up to 250-fold.

stat.ME↗

arXiv2026

NIRVAR: Network Informed Restricted Vector Autoregression

High-dimensional panels of time series often arise in finance and macroeconomics, where co-movements within groups of panel components occur. Extracting these groupings from the data provides a coarse-grained description of the complex system in question and can inform subsequent prediction tasks. We develop a novel methodology to model such a panel as a restricted vector autoregressive process, where the coefficient matrix is the weighted adjacency matrix of a stochastic block model. This network time series model, which we call the Network Informed Restricted Vector Autoregression (NIRVAR) model, yields a coefficient matrix that has a sparse block-diagonal structure. We propose an estimation procedure that embeds each panel component in a low-dimensional latent space and clusters the embedded points to recover the blocks of the coefficient matrix. Crucially, the method allows for network-based time series modelling when the underlying network is unobserved. We derive the bias, consistency and asymptotic normality of the NIRVAR estimator. Simulation studies suggest that the NIRVAR estimated embedded points are Gaussian distributed around the ground truth latent positions. On three applications to finance, macroeconomics, and transportation systems, NIRVAR outperforms competing models in terms of prediction and provides interpretable results regarding group recovery.

stat.ME↗