Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2511.03769

Current validation practice undermines surgical AI development

Annika Reinke·Ziying O. Li·Minu D. Tizabi·Pascaline André·Marcel Knopp·Mika M. Rother·Ines P. Machado·Maria S. Altieri·Deepak Alapatt·Sophia Bano·Sebastian Bodenstedt·Oliver Burgert·Elvis C. S. Chen·Justin W. Collins·Olivier Colliot·Evangelia Christodoulou·Tobias Czempiel·Adrito Das·Reuben Docea·Daniel Donoho·Qi Dou·Jennifer Eckhoff·Sandy Engelhardt·Gabor Fichtinger·Philipp Fuernstahl·Pablo García Kilroy·Stamatia Giannarou·Stephen Gilbert·Ines Gockel·Patrick Godau·Jan Gödeke·Teodor P. Grantcharov·Tamas Haidegger·Alexander Hann·Makoto Hashizume·Charles Heitz·Rebecca Hisey·Hanna Hoffmann·Arnaud Huaulmé·Paul F. Jäger·Pierre Jannin·Anthony Jarc·Rohit Jena·Yueming Jin·Leo Joskowicz·Luc Joyeux·Max Kirchner·Axel Krieger·Gernot Kronreif·Kyle Lam·Shlomi Laufer·Joël L. Lavanchy·Gyusung I. Lee·Robert Lim·Peng Liu·Lucas Luttner·Hani J. Marcus·Pietro Mascagni·Leon Mayer·Ozanan R. Meireles·Beat P. Mueller·Lars Mündermann·Hirenkumar Nakawala·Nassir Navab·Abdourahmane Ndong·Juliane Neumann·Felix Nickel·Marco Nolden·Chinedu Nwoye·Namkee Oh·Nicolas Padoy·Thomas Pausch·Micha Pfeiffer·Tim Rädsch·Hongliang Ren·Nicola Rieke·Dominik Rivoir·Duygu Sarikaya·Samuel Schmidgall·Matthias Seibold·Silvia Seidlitz·Alexander Seitel·Lalith Sharan·Jeffrey H. Siewerdsen·Vinkle Srivastav·Raphael Sznitman·Russell Taylor·Thuy N. Tran·Matthias Unberath·Fons van der Sommen·Martin Wagner·Amine Yamlahi·Shaohua K. Zhou·Aneeq Zia·Amin Madani·Danail Stoyanov·Stefanie Speidel·Daniel A. Hashimoto·Fiona R. Kolbinger·Lena Maier-Hein
View original source ↗Read source document ↗Export metadata ↓

Abstract

Surgical data science (SDS) is rapidly advancing, yet clinical adoption of artificial intelligence (AI) in surgery remains limited, with inadequate validation as an important contributing factor. Existing validation practices often neglect the temporal and hierarchical structure of intraoperative videos, yielding misleading or clinically irrelevant results. We introduce a comprehensive catalogue of validation pitfalls in AI-based surgical video analysis, derived from a multi-stage Delphi process with 92 international experts. Pitfalls span three categories: (1) data, (2) metric selection/configuration, and (3) aggregation and reporting. A systematic review of surgical AI papers reveals that these pitfalls are widespread. Experiments on surgical video datasets show that ignoring temporal and hierarchical data structures can understate uncertainty, obscure critical failure modes, and alter algorithm rankings. To address these shortcomings, we provide consensus-based best practices compiled. Together, this work provides an evidence-based framework for rigorous validation of surgical video analysis algorithms, guiding benchmarking, reporting, regulatory review, and clinical translation.

Explore related subjects

q-bio.OT

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Annika Reinke, Ziying O. Li, Minu D. Tizabi, Pascaline André, Marcel Knopp, Mika M. Rother, Ines P. Machado, Maria S. Altieri, Deepak Alapatt, Sophia Bano, Sebastian Bodenstedt, Oliver Burgert, Elvis C. S. Chen, Justin W. Collins, Olivier Colliot, Evangelia Christodoulou, Tobias Czempiel, Adrito Das, Reuben Docea, Daniel Donoho, Qi Dou, Jennifer Eckhoff, Sandy Engelhardt, Gabor Fichtinger, Philipp Fuernstahl, Pablo García Kilroy, Stamatia Giannarou, Stephen Gilbert, Ines Gockel, Patrick Godau, Jan Gödeke, Teodor P. Grantcharov, Tamas Haidegger, Alexander Hann, Makoto Hashizume, Charles Heitz, Rebecca Hisey, Hanna Hoffmann, Arnaud Huaulmé, Paul F. Jäger, Pierre Jannin, Anthony Jarc, Rohit Jena, Yueming Jin, Leo Joskowicz, Luc Joyeux, Max Kirchner, Axel Krieger, Gernot Kronreif, Kyle Lam, Shlomi Laufer, Joël L. Lavanchy, Gyusung I. Lee, Robert Lim, Peng Liu, Lucas Luttner, Hani J. Marcus, Pietro Mascagni, Leon Mayer, Ozanan R. Meireles, Beat P. Mueller, Lars Mündermann, Hirenkumar Nakawala, Nassir Navab, Abdourahmane Ndong, Juliane Neumann, Felix Nickel, Marco Nolden, Chinedu Nwoye, Namkee Oh, Nicolas Padoy, Thomas Pausch, Micha Pfeiffer, Tim Rädsch, Hongliang Ren, Nicola Rieke, Dominik Rivoir, Duygu Sarikaya, Samuel Schmidgall, Matthias Seibold, Silvia Seidlitz, Alexander Seitel, Lalith Sharan, Jeffrey H. Siewerdsen, Vinkle Srivastav, Raphael Sznitman, Russell Taylor, Thuy N. Tran, Matthias Unberath, Fons van der Sommen, Martin Wagner, Amine Yamlahi, Shaohua K. Zhou, Aneeq Zia, Amin Madani, Danail Stoyanov, Stefanie Speidel, Daniel A. Hashimoto, Fiona R. Kolbinger, Lena Maier-Hein. 2026-07-31. Current validation practice undermines surgical AI development. https://arxiv.org/abs/2511.03769

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Flash-Radiomics: A Scalable Hybrid CPU-CUDA Engine for Standardized Scalar Radiomics and Accelerated Spatial Mapping

Background and Objectives: Spatial mapping retains the spatial distribution of radiomic features, but computational cost and fragmented software limit its use. We developed Flash-Radiomics with scalar extraction and spatial mapping, a central processing unit (CPU) backend, a hybrid Compute Unified Device Architecture (CUDA) backend, consistent feature names, and Hierarchical Data Format version 5 (HDF5) storage. Methods: We evaluated Image Biomarker Standardisation Initiative (IBSI) compliance, CPU-CUDA concordance, and end-to-end processing time. Compliance testing included 825 chapter 1 (IBSI-1) tests covering 165 high-consensus features and 323 chapter 2 (IBSI-2) tests with numerical references. Concordance testing included 1,148 scalar pairs and 93 spatial-map pairs. End-to-end processing time was measured five times per input volume of interest (VOI) size. Comparisons included the Medical Image Radiomics Processor (MIRP) and PyRadiomics for 102 shared scalar features and PyRadiomics for 93 shared spatial maps. Results: Both backends passed all 1,148 IBSI tests, and all paired results were concordant. At the largest scalar input, CPU required 76.343 s and hybrid CUDA 81.915 s; CPU was 4.7 times faster than MIRP and 190.7 times faster than PyRadiomics. At the largest spatial input completed by both backends, hybrid CUDA reduced processing time by 68.6% relative to CPU (79.280 versus 252.791 s). At PyRadiomics' largest completed spatial input, hybrid CUDA was 84.9 times faster. Conclusions: Flash-Radiomics unified standardized scalar extraction, spatial mapping, concordant CPU-CUDA results, and HDF5 storage. CPU processing time was similar or shorter for scalar extraction, whereas hybrid CUDA was faster for spatial mapping under the tested conditions.

q-bio.OT↗
arXiv2026

EPI-KAN: A Method For Estimating and Forecasting Time-Dependent COVID-19 Parameters

We introduce EPI-KAN, a novel method for estimating COVID-19 time-varying parameters. EPI-KAN uses historical epidemiological data, Physics-Informed Neural Network (PINN), and the novel Kolmogorov-Arnold Network (KAN). The method harnesses the novel Kolmogorov-Arnold Network (KAN), which is a type of artificial neural network. For the KAN in this paper, we learn activation functions that are represented using Fourier series, hence we abbreviate as KAN-F. In this study, we estimate parameters in the context of an SIRD compartmental differential equations. The time-dependent parameters are the transmission rate $β(t)$, recovery rate $γ(t)$, and mortality rate $μ(t)$. We define three KAN-F functions $\widehatβ$, $\widehatγ$, $\widehatμ$ that model the true parameters $β(t)$, $γ(t)$, $μ(t)$, respectively. We test two model architectures for the KAN-F: the first has 8 input variables consisting of $S$, $I$, $R$, $D$, and their numerical gradients at any time $t$, while the second has 4 input variables excluding the numerical gradients. The objective loss function that has to be minimized is subject to Physics-Informed Neural Network (PINN). Using historical data of COVID-19 from three South-East Asian countries: Indonesia, Singapore, and Malaysia, we are able to estimate $β(t)$, $γ(t)$, and $μ(t)$ on each country with decent accuracy and efficiency. The time period of choice coincides with the period where SARS-CoV-2 Delta variant (B.1.617.2) was dominant. In addition to estimating the rates during the training period, we also predict transmission rates over 30 days during forecast period. We found that the output of KAN-F over the forecast period can give good predictions if we scale the output by a factor of 17\% for Indonesia and 30\% for Singapore and Malaysia.

q-bio.OT↗
arXiv2026

Making Models That Matter: How to Build Trustworthy and Useful Systems Biology Models

Computational models supporting mechanistic understanding of (complex) biological systems, systems behaviour prediction, and experimental design are becoming more and more embedded in research on complex biological systems. Reuse and refinement of models, rather than continuous reinvention, is becoming increasingly important as models' demands on computational infrastructure increase. However published models - despite the variety of efforts taken so far - are frequently difficult to reproduce or reuse, substantially limiting their scientific value. Here we address the requirements for model reusability in the light of the field-specific CURE framework (Credible, Understandable, Reproducible, Extensible) and the more general FAIR principles (Findable, Accessible, Interoperable, Reusable). Considering published guidance we identify broad agreement on requirements for findability, accessibility, and interoperability, but continued lack of clarity and consensus around reusability. Focusing on the scientific quality and usability of computational models we discuss six key practices underpinning model sharing and re-use. Mapping the FAIR and CURE principles onto the model lifecycle we propose ten recommendations for building and sharing systems biology models that are both FAIR- and CURE-compliant.

q-bio.OT↗