A repeated k-fold cross-validation approach for evaluating the instability of clinical prediction models: an empirical comparison to the bootstrap approach

Nop Khongthon; Pakpoom Wongyikul; Noraworn Jirattikanwong; Phanu Prasankittirach; Natthanaphop Isaradech; Wachiranun Sirikul; Noppadon Seesuwan; Suppachai Lawanaskol; Phichayut Phinyo

arXiv · 2607.03676

A repeated k-fold cross-validation approach for evaluating the instability of clinical prediction models: an empirical comparison to the bootstrap approach

Nop Khongthon·Pakpoom Wongyikul·Noraworn Jirattikanwong·Phanu Prasankittirach·Natthanaphop Isaradech·Wachiranun Sirikul·Noppadon Seesuwan·Suppachai Lawanaskol·Phichayut Phinyo

View original source ↗Read source document ↗Export metadata ↓

Abstract

Bootstrap-based methods have been recommended for assessing prediction instability in clinical prediction models, but their performance relative to cross-validation (CV) remains unclear. We propose a CV-based approach for assessing prediction instability and compare it with a bootstrap-based approach in logistic regression and random forest models. We conducted a resampling-based empirical experiment using a clinical cohort of 19,418 emergency department patients. Development samples were generated under events-per-variable (EPV) scenarios of 10, 30, and 50, and results were compared with those from the full dataset. Models were evaluated using bootstrap validation and repeated 5-fold CV; nested CV was used for random forest tuning. Predictive performance was assessed using AUC, calibration slope, and calibration-in-the-large. Prediction instability was quantified using mean absolute prediction error (MAPE). For logistic regression, bootstrap validation and repeated 5-fold CV produced broadly similar discrimination and calibration, especially at higher EPV values. For random forest, apparent performance consistently overestimated empirical discrimination. Bootstrap validation and repeated 5-fold CV gave comparable discrimination, but repeated 5-fold CV produced calibration slope estimates closer to the empirical value. Prediction stability improved as EPV increased for both modelling approaches. At EPV 30, bootstrap-derived MAPE was higher than CV-derived MAPE for both logistic regression (median, 0.042 versus 0.020) and random forest (median, 0.077 versus 0.027). A CV-based approach can assess prediction instability while also providing internally validated performance. These findings support CV-based instability assessment as a practical alternative to bootstrap-based assessment, particularly when comparing instability across multiple modelling algorithms.

Explore related subjects

stat.ME

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Nop Khongthon, Pakpoom Wongyikul, Noraworn Jirattikanwong, Phanu Prasankittirach, Natthanaphop Isaradech, Wachiranun Sirikul, Noppadon Seesuwan, Suppachai Lawanaskol, Phichayut Phinyo. 2026-07-04. A repeated k-fold cross-validation approach for evaluating the instability of clinical prediction models: an empirical comparison to the bootstrap approach. https://arxiv.org/abs/2607.03676

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Efficient and scalable clustering of survival curves

Survival analysis encompasses a broad range of methods for analyzing time-to-event data, with one key objective being the comparison of survival curves across groups. Traditional approaches for identifying clusters of survival curves often rely on computationally intensive bootstrap techniques to approximate the null hypothesis distribution. While effective, these methods impose significant computational burdens. In this work, we propose a novel approach that leverages the k-means and log-rank test to efficiently identify and cluster survival curves. Our method eliminates the need for computationally expensive resampling, significantly reducing processing time while maintaining statistical reliability. By systematically evaluating survival curves and determining optimal clusters, the proposed method ensures a practical and scalable alternative for large-scale survival data analysis. Through simulation studies, we demonstrate that our approach achieves results comparable to existing bootstrap-based clustering methods while dramatically improving computational efficiency. These findings suggest that the log-rank-based clustering procedure offers a viable and time-efficient solution for researchers working with multiple survival curves in medical and epidemiological studies.

stat.ME↗

arXiv2026

Bayesian Modular Inference for Copula Models with Potentially Misspecified Marginals

Copula models of multivariate data are popular because they allow separate specification of marginal distributions and the copula function. These components can be treated as inter-related modules in a modified Bayesian inference approach called ''cutting feedback'' that is robust to their misspecification. Recent work uses a two module approach, where all $d$ marginals form a single module, to robustify inference for the marginals against copula function misspecification, or vice versa. However, marginals can exhibit differing levels of misspecification, making it attractive to assign each its own module with an individual influence parameter controlling its contribution to a joint semi-modular inference (SMI) posterior. This generalizes existing two module SMI methods, which interpolate between cut and conventional posteriors using a single influence parameter. We develop a novel copula SMI method and select the influence parameters using Bayesian optimization. It provides an efficient continuous relaxation of the discrete optimization problem over $2^d$ cut/uncut configurations. We establish theoretical properties of the resulting semi-modular posterior and demonstrate the approach on simulated and real data. The real data application uses a skew-normal copula model of asymmetric dependence between equity volatility and bond yields, where robustifying copula estimation against marginal misspecification is strongly motivated.

stat.ME↗

arXiv2026

On Asymptotic Outlier Rejection in Bayesian Mixed Poisson Regression Models Under Extreme Target and Covariate Values

Bayesian models are defined to be fully robust against outliers if observations infinitely far from the other data do not influence the posterior. In regression models, this entails a need to consider outliers in both target and covariate values. While in linear regression these cases are interchangeable, as both lead to anomalously large residuals, this symmetry does not apply to generalized linear models. Importantly, Hamura et al. (2025, arXiv:2106.10503) presented sufficient conditions for mixed Poisson count regression models to be robust against infinitely large target values and proposed a mixed Poisson-Rescaled Beta model fulfilling these conditions. We continue from their work and study the robustness properties of mixed Poisson regression models with Gaussian latent variables in the presence of outliers in covariates. We show that in count regression the symmetry between covariate and target outliers breaks: mixed Poisson models are not robust to outlier covariates even if they were robust to target outliers. Furthermore, we show that, as a covariate gets infinitely large, the corresponding regression coefficient posterior collapses to a point-mass distribution concentrated around zero. We hence introduce a novel robustified mixed Poisson model which we denote as $x$-outlier rejective ($x$-OR) mixed Poisson, and demonstrate its theoretical and practical ability to accommodate outliers in the covariates. We investigate robustness properties of alternative ($x$-OR) mixed Poisson models in the presence of moderate outliers with simulations and a real world case study and show that the $x$-OR log-$t$ and Rescaled Beta mixed Poisson models reject both types of outliers, leading to improved inference.

stat.ME↗