On the Benefit of Blocking for Online Experiments with Skewed Data

Abbas Zaidi; Matthew Reimherr; Rina Friedberg; Fabian Martinez; Richard Mudd

Home/Archive/arXiv

arXiv · 2610.10435

On the Benefit of Blocking for Online Experiments with Skewed Data

Abbas Zaidi·Matthew Reimherr·Rina Friedberg·Fabian Martinez·Richard Mudd

View original source ↗Read source document ↗Export metadata ↓

Abstract

This paper advocates for using blocked (stratified) assignment in online A/B tests to handle highly skewed population data. While blocking yields only modest precision gains (5-10%) due to the limits of discretization, the authors demonstrate it offers two crucial structural benefits over post-hoc statistical adjustments. First, fixed-weight blocking correctly targets the true average treatment effect, avoiding the severe bias introduced by efficiency-weighted alternatives. Second, blocking localizes extreme outliers, enabling targeted within-block winsorization that effectively controls noise without destroying the tail-end treatment effect.

Explore related subjects

stat.AP

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Abbas Zaidi, Matthew Reimherr, Rina Friedberg, Fabian Martinez, Richard Mudd. 2026-10-07. On the Benefit of Blocking for Online Experiments with Skewed Data. https://arxiv.org/abs/2610.10435

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Evaluating LiDAR Data Sources, Predictor Resolution, and Spatial Random Effects in Bayesian Change-of-Support Models for Forest Inventory

Forest managers need timely stand-level information to support operational planning, particularly in mixed-species, structurally heterogeneous forests facing climate-related disturbance. Model-based estimation combines sparse field data with remotely sensed predictors to estimate growing stock volume (GSV) for small areas. While uncrewed aerial vehicle laser scanning (ULS) offers flexible, high-resolution LiDAR acquisition, its advantages over conventional airborne laser scanning (ALS) remain unclear. We compared publicly available ALS and newly acquired ULS data using Bayesian change-of-support models to estimate GSV in a mixed-species forest in north-eastern Germany. We evaluated distributional LiDAR metrics and spatial random effects. ULS consistently outperformed ALS, achieving cross-validated RMSPEs of 68.5 m^3/ha and 79.5 m3/ha, respectively-a 13.8 % reduction in prediction error. Distributional metrics improved ULS models more strongly, reducing RMSPE by up to 10.1 %; spatial effects provided only minor gains at substantially higher computational cost. ULS also produced lower uncertainty in latent stand-mean GSV estimates. The ULS advantage may reflect both finer-scale canopy information and closer temporal alignment with field measurements. Timely, information rich LiDAR may therefore be more valuable for stand-level GSV estimation than increasingly complex spatial models. Temporally matched ALS-ULS comparisons are needed to isolate platform effects.

stat.AP↗

arXiv2026

DeepAJM: Deep Association Joint Model for Irregularly Sampled data

Joint Models simultaneously model longitudinal and survival outcomes, leveraging patterns in patients' longitudinal trajectory to improve the prediction of survival outcomes. The classical parametric joint models, however, rely on fixed parametric assumptions, making them susceptible to bias under model misspecification and smaller sample sizes. We propose a deep joint model, DeepAJM, that does not require any parametric assumptions, while retaining a partially interpretable, per-longitudinal-outcome association structure. The joint model uses an encoder-decoder (sequence-to-sequence) architecture to learn the latent structure in patients' time-varying covariate trajectories. The model links the longitudinal processes to the survival processes through a learned interpretable association structure, in which each longitudinal output from the decoder gets remodulated by baseline covariates before it contributes to the risk scores from the survival head of the architecture. The model was evaluated on three datasets ( a cardiovascular-disease EHR cohort, a primary biliary cirrhosis (PBC2) dataset, and a simulated dataset) against a classical parametric joint model, TransformerJM, DA-LSTM and a Cox-based survival-only model. All models were assessed using C-index, integrated brier score (IBS), time-dependent AUROC, and time-dependent AUPRC. Our model achieved the best discrimination in terms of the C-index, time-dependent AUROC, and AUPRC across all datasets.

stat.AP↗

arXiv2026

Bayesian Optimization for Dose Finding with Two Agents: Participant Allocation and Final Selection

In two-agent dose-finding trials, the next cohort should help identify a combination for final selection. We studied a constrained knowledge-gradient (cKG) rule with one-cohort lookahead that updates independent Gaussian-process models of efficacy and continuous toxicity, reapplies a probability criterion for mean toxicity, and evaluates the resulting selection. We derived a deterministic calculation over a fixed set of dose combinations, holding fitted model parameters fixed during each hypothetical update. We compared cKG with constrained expected improvement (cEI) and two toxicity-only rules, targeted mean squared error (tMSE) and entropy, in four synthetic scenarios. In the primary obstructive sleep apnea (OSA)-derived scenario, averaged equally over strata and five probability cutoffs, cKG assigned fewer participants to combinations above the true mean-toxicity limit than tMSE (17.92% versus 27.08%), but selected such combinations more often at trial completion (18.80% versus 11.85%). Compared with cEI, cKG had higher mean simulated reduction in the 4%-desaturation apnea-hypopnea index (AHI4) at final selection (7.46 versus 6.72 events/hour), more above-limit final selections (18.80% versus 10.50%), and more above-limit assignments (17.92% versus 15.10%). Across scenarios, its efficacy advantage over cEI was smaller under stricter toxicity criteria. Continuous outcomes, uncalibrated toxicity limits, and a rule that still selects a combination when none meets the criterion limit clinical interpretation. Allocation and final-selection toxicity should be reported separately, alongside efficacy.

stat.AP↗