RetroHolmes: When Semantic Plausibility Fails Retrospective Physical Process Reasoning
Vision-Language Models (VLMs) are widely used for visual understanding, yet current evaluation protocols fail to assess whether these capabilities are grounded in physical reasoning. To address this gap, we introduce Retrospective Physical Process Reasoning, a new evaluation paradigm to reason backward from outcomes under explicit physical constraints. Building on the paradigm, we present RetroHolmes, the first real-world benchmark for Retrospective Physical Process Reasoning, comprising object-centric image pairs annotated with reachability labels and causal step sequences across diverse physical transitions. Using RetroHolmes, we analyze VLMs and uncover systematic failure modes, including judgment bias in reachability assessment and belief dominance over physical evidence, mirroring sycophancy behavior observed in large language models. Our quantitative analyses link these failures to reliance on linguistic priors and attention concentrated on visually invariant regions, suggesting limited physical simulation of the intermediate states connecting visual endpoints. To address these limitations, we propose Simulate-and-Verify, an analysis-by-synthesis framework that grounds reachability judgment and step reconstruction in visual simulation. Experiments show that Simulate-and-Verify improves judgment accuracy by 21.67 percentage points and reduces belief dominance by 10.39 percentage points compared with GPT-5.5, demonstrating the effectiveness of visual simulation in grounding physical reasoning.