The Illusion of Improvement: Reject Inference Strategies in Credit Scoring

Abstract

Reject inference methods are widely used to mitigate survival bias in credit scoring, yet their effectiveness remains poorly understood. We systematically evaluate several such methods and uncover a structural failure mode: in a natural retraining cycle, models whose accuracy improves while recall collapses create an illusion of improvement that leads practitioners to believe the system is getting better when, in fact, its rejection quality---the ability to correctly screen out defaulters---is deteriorating. We then propose a controlled exploration strategy that breaks the feedback loop without statistical assumptions: the lender deliberately approves a fraction of rejected applicants and observes their true outcomes. We show that accuracy and rejection quality respond in opposite directions to exploration: accuracy is lower under full exploration than under no exploration, while rejection quality is higher. A lender optimizing accuracy on the pool it has itself assembled finds no exploration rate that improves it and will therefore not explore, which is the worst choice for learning. Even minimal exploration rates (2\%--5\%) prove sufficient in our experiments to diagnose the severity of the feedback loop at near-zero cost. Our findings are consistent across two machine learning methods and three real-world datasets, and suggest that standard evaluation protocols are inadequate for assessing models trained under survival bias.