arXiv · 2610.34464
Retrospective Comparison of Radiation Oncology Patient Message Responses Generated by Epic and Institutionally Managed LLMs
Abstract
We retrospectively compared Epic-generated draft replies with institutionally managed hybrid GPT-4.1 and Gemini 2.5 Pro responses to 51 selected single-turn radiation oncology messages. Six clinicians, blinded to response source, rated overall quality on a five-point scale and perceived editing effort on a four-point scale. Mixed-effects models accounted for repeated ratings by message and clinician. Adjusted overall-quality means were 2.95 for Epic, 4.27 for hybrid GPT-4.1, and 4.39 for hybrid Gemini 2.5 Pro. Corresponding editing-effort means were 2.38, 1.46, and 1.41, with lower scores indicating less perceived effort. Both hybrid models received more favorable ratings than Epic for both outcomes (all Holm-adjusted P < 0.001), without a statistically significant difference between the hybrid models. These findings support prospective evaluation of clinician-supervised draft generation, though editing time and patient outcomes were not measured, and clinical safety was not established.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Meiyun Cao, Federico Mastroleo, Shuoyang Wei, Yuexing Hao, Srinivas Seetamsetty, Keldon K. Lin, Madhurya Reddy, Yuzhen Ding, Jason Holmes, Jonathan E. Moonen, Shyanne L. Dobbs, Diego Santos Toesca, Mark Waddle, Wei Liu. 2026-10-25. Retrospective Comparison of Radiation Oncology Patient Message Responses Generated by Epic and Institutionally Managed LLMs. https://arxiv.org/abs/2610.34464
Cite the original work for its findings. Save a collection to share your selection of sources.