Evaluation of clinical prediction models: Decomposing performance uncertainty into model development and validation components

Abstract

There has been growing emphasis on acknowledging, quantifying, and communicating the uncertainty inherent in the output of clinical prediction models. However, allowing for model uncertainty when assessing the performance of an existing model in a new sample (i.e., external validation) has not been sufficiently explored. When validating a model that generates uncertain predictions, the uncertainty of a given performance metric is affected by both model uncertainty and the finiteness of the validation sample (validation uncertainty). For a given performance metric, we show that the total uncertainty of the performance estimate can be decomposed into the uncertainty due to the model and the uncertainty due to the validation. Also, we propose $M^2$, the fraction of the total variance due to model uncertainty, and $V^2$, the fraction of the total variance due to validation uncertainty. These measures serve as complementary scalar indices that can be implemented for common performance metrics and visualizations such as calibration plots using simple Monte Carlo algorithms. Critically, our case study shows that with the same model and the same validation sample, the contribution of uncertainty sources is highly variable across performance metrics ($M^2$ varying from 12% for the Brier score to 78% for the calibration slope). This indicates that stakeholders who prioritize different aspects of performance (e.g., calibration versus clinical utility) might arrive at different decisions on whether future efforts should be directed towards creating a more reliable model or procuring a larger validation sample. In conclusion, external validation studies can readily be amended to incorporate model uncertainty, and we recommend that such studies report total variance as well as the relative share of variance components for all performance metrics.