SIGHT: A Methodological Appraisal Framework for Evaluating Large Language Models in Ophthalmology

Abstract

Large language models (LLMs) are increasingly evaluated across ophthalmic tasks, yet the methodological quality of these evaluations has received little attention. Most studies report accuracy while overlooking design features required for valid and clinically meaningful inference. We critically reviewed published ophthalmic LLM evaluation studies covering knowledge assessment, patient education, diagnostic reasoning, triage, clinical documentation, multimodal image interpretation, and uncertainty estimation. We used the methodological weaknesses that recurred across study designs to develop the SIGHT framework, which has five domains: Sample and item provenance, Independence and blinding, Ground truth and comparator fairness, Handling of paired and repeated-measures data, and Transparency and trustworthiness. Current evaluations rely heavily on benchmark accuracy, synthetic clinical vignettes, and curated datasets, and real-world clinical validation remains rare. Common limitations include potential training-data contamination, inadequate handling of paired data, limited assessment of inter-rater reliability, inconsistent reference standards, insufficient evaluation of calibration and uncertainty, and frequent overinterpretation of benchmark performance as evidence of clinical readiness. SIGHT provides twenty structured appraisal items to evaluate internal validity, interpretability, and applicability across diverse ophthalmic LLM study designs. It offers a practical way to appraise ophthalmic LLM studies and may improve the methodological quality, interpretability, and clinical relevance of future evaluations.