PUID: A Personalized Deconfounding Framework for Recommender Systems under Hidden Confounding

Abstract

Recommender systems often rely on observational user-item interaction data, which is prone to selection bias due to users' selective interactions with items. While techniques such as inverse propensity weighting (IPW) and doubly robust estimators are effective in addressing selection bias from observed confounding, they become unreliable when hidden confounding exists, meaning that there are confounders that influence both user clicks and feedback but are not observable (e.g., user salary). Existing approaches relying on randomized controlled trials (RCTs) or global sensitivity bounds are constrained in practice: RCTs demand costly experimental data, while global sensitivity bounds presume a uniformly bounded effect of unmeasured confounders on propensities through sensitivity analysis, thereby neglecting heterogeneity across user-item interactions. To overcome this limitation, we propose a novel framework, Personalized Unobserved-Confounding-aware Interaction Deconfounder (PUID), which estimates user-item level sensitivity bounds, thereby substantially relaxing the homogeneity assumption inherent in global sensitivity bounds. Under mild assumptions, we use an entropy-based method to estimate the individualized strength of hidden confounding. Specifically, if observed user and item features can already well predict the exposure status (i.e., the mutual information is small), then the influence of hidden confounding is assumed to be small. To ensure both robustness and predictive accuracy, we further develop an adversarial optimization strategy and propose a benchmark-guided variant (BPUID) that incorporates pre-trained models as stabilizing references. Extensive experiments on three real-world datasets demonstrate that our approach consistently outperforms state-of-the-art baselines under hidden confounding, without requiring RCT data.