Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models

Abstract

Vision-language models (VLMs) are increasingly applied to medical visual question answering (Med-VQA), yet whether they can \emph{localize} the evidence behind their answers---a prerequisite for clinical auditability---is poorly characterized. We separately evaluate VQA reasoning and visual grounding for four recent frontier VLMs (GPT-5.1, GPT-5.5, Gemini-2.5-Pro, Gemini-3-Flash), two domain-specific medical VLMs (Lingshu, MedGemma), and a dedicated open-vocabulary detector (Grounding DINO) on VQA-RAD and SLAKE. Two findings challenge the intuition that ``add grounding to improve VQA.'' First, \textbf{no model localizes medical targets well}: every off-the-shelf system---frontier, medical-specialized, or dedicated detector---scores mean IoU 0.05--0.24 on our SLAKE grounding split, at or barely above a trivial center-box baseline (0.10), with Acc@0.5 below 20\%. Second, and counter to the common ``localize-then-answer'' paradigm, \textbf{cropping to a bounding box degrades VQA even when the box is a perfect oracle}: on the matched subset where oracle ground-truth boxes are applied, GT-grounding \emph{lowers} closed-ended accuracy for every model (by 0.9--18.0 points versus using the full image)---consistent with the crop discarding global context the model relies on. Because the oracle box removes localization error by construction, the problem is not that perception is a recoverable bottleneck, but that grounding-by-cropping is itself the wrong interface. Finally, we show constructively that the two channels need not conflict: supervised fine-tuning of Qwen-2.5-VL-7B on answers \emph{alone} silently destroys box-evidence emission (0/418 parseable boxes), whereas mixing in a small amount of grounding supervision restores localization to 0.36 IoU---above every zero-shot model---while preserving answer accuracy.