Transfer Learning under Covariate Shift: Local $k$-Nearest Neighbours Regression with Heavy-Tailed Design

Abstract

Covariate shift is a common transfer learning scenario where the marginal distributions of input variables vary between source and target data while the conditional distribution of the output variable remains consistent. The existing notions describing differences between marginal distributions face limitations in handling scenarios with unbounded support, particularly when the target distribution has a heavier tail. To overcome these challenges, we introduce a new concept called density ratio exponent to quantify the relative decay rates of marginal distributions' tails under covariate shift. Furthermore, we propose the local k-nearest neighbour regressor for transfer learning, which adapts the number of nearest neighbours based on the marginal likelihood of each test sample. From a theoretical perspective, convergence rates with and without supervision information on the target domain are established. Those rates indicate that our estimator achieves faster convergence rates when the density ratio exponent satisfies certain conditions, highlighting the benefits of using density estimation for determining different numbers of nearest neighbours for each test sample. Our contributions enhance the understanding and applicability of transfer learning under covariate shift, especially in scenarios with unbounded support and heavy-tailed distributions.