Why Machine Learning Models Systematically Underestimate Extreme Values

Abstract

A persistent challenge in astronomical machine learning is a systematic bias where predictions compress the dynamic range of true values-high values are consistently predicted too low while low values are predicted too high. Understanding this bias has important consequences for astronomical measurements and our understanding of physical processes in astronomical inference. Through analytical examination of linear regression, we show that this bias arises naturally from measurement uncertainties in input features and persists regardless of training sample size, label accuracy, or parameter distribution. In the univariate case, we demonstrate that attenuation becomes important when the ratio of intrinsic signal range to measurement uncertainty ($σ_{\text{range}}/σ_x$) is below $O(10)$-a regime common in astronomy. We further extend the theoretical framework to multivariate linear regression and demonstrate its implications using stellar spectroscopy as a case study. Even under optimal conditions-high-resolution APOGEE-like spectra ($R=24,000$) with high signal-to-noise ratios (SNR=100) and multiple correlated features-we find percent-level bias. The effect becomes even more severe for modern-day low-resolution surveys like LAMOST and DESI due to the lower SNR and resolution. These findings have broad implications, providing a theoretical framework for understanding and addressing this limitation in astronomical data analysis with machine learning.