Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2607.16595

Orientation Reading by Production Vision-Language Models on Optotype Charts: A Controlled Multi-Model Evaluation Across Reasoning Modes, Prompts, and Access Modalities

Shahryar Wasif·Avneek Sandhu·Bin Hu
View original source ↗Read source document ↗Export metadata ↓

Abstract

OBJECTIVES: Vision-language models are increasingly used to interpret medical and everyday images through consumer chat interfaces, yet their ability to read orientation - the single perceptual operation tested by the tumbling-E acuity optotype - is poorly characterized on the surfaces through which they are actually used. METHODS: We evaluated four production vision-language models (referred to as Claude, GPT, GROK, and Gemini) through their consumer chat interfaces on a locked set of seven optotype charts: four uniform tumbling-E charts (one per cardinal orientation), two mixed-orientation tumbling-E charts, and one Snellen letter chart as a specificity control. Each model was run in two reasoning modes (Fast and Thinking) under two prompt variants (with and without an explicit orientation-decoding rule) by up to three operators. The corpus comprised 920 scoreable trials and 50,420 glyph judgements. The primary outcome was glyph-level accuracy against the chart's designed orientation, summarized with Wilson 95% confidence intervals. RESULTS: Accuracy ranged from 43.0% to 97.0% across models on identical charts, and the strongest model depended on reasoning mode (GPT 97.0% in Fast mode; GROK 96.6% in Thinking mode). Errors were not random but collapsed onto a model-specific attractor direction. Models were 96-100% internally self-consistent yet ranged widely in accuracy, dissociating reliability from validity. An answer-key-free ensemble-consensus estimate tracked accuracy closely (r = 0.998). For one model, consumer-interface accuracy fell 25-27 points below programmatic access, almost entirely on a single orientation. CONCLUSIONS: A single accuracy figure conceals clinically relevant, orientation-specific failure modes; vision-language models should be evaluated along multiple axes and on the deployment surface before image-interpretation outputs are trusted.

Explore related subjects

q-bio.OT

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Shahryar Wasif, Avneek Sandhu, Bin Hu. 2026-07-18. Orientation Reading by Production Vision-Language Models on Optotype Charts: A Controlled Multi-Model Evaluation Across Reasoning Modes, Prompts, and Access Modalities. https://arxiv.org/abs/2607.16595

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

2D reaction-diffusion model-based biopsy simulation for dynamic tumor growth parameter estimation

Once diagnosed, cancer requires a fast, reliable and preferably cost-efficient assessment of the current state and potential progression of the disease. A new method for estimating tumor cell diffusivity $D$ and proliferation rate $γ$ in the context of the mechanistic reaction-diffusion equation from single-point-in-time routine biopsies aims to deliver just that, and quantities computed from the parameter estimates have recently been tested as a new biomarkers for risk-stratification in radiotherapy. Here, we extend the findings of this previous work by providing a first theoretical validation. The method is applied to in-silico biopsies which are generated by solving the two-dimensional reaction-diffusion equation for different growth terms (exponential and logistic) with a Dirac-Delta initial condition, and transforming the continuous results into spatial point patterns via a form of reverse coarse-graining. If no information about tumor age is used, in short-term experiments the original dispersion length $\sqrt{D/γ}$ could be retrieved with a relative root mean squared error (RRMSE) of around 8% and an $\text{R}^2$-value of 0.97. In long-term experiments, the RRMSEs ranged from 8 to 14% and the $\text{R}^2$-values from 0.75 to 0.98. The scaled front velocity $\sqrt{D \cdotγ}$, which can only be estimated if information about tumor age is available, was retrieved with an RRMSE of 7% and an $\text{R}^2$ of 0.98 in both, the short-term and the long-term experiments.

q-bio.OT↗
arXiv2026

Flash-Radiomics: A Scalable Hybrid CPU-CUDA Engine for Standardized Scalar Radiomics and Accelerated Spatial Mapping

Background and Objectives: Spatial mapping retains the spatial distribution of radiomic features, but computational cost and fragmented software limit its use. We developed Flash-Radiomics with scalar extraction and spatial mapping, a central processing unit (CPU) backend, a hybrid Compute Unified Device Architecture (CUDA) backend, consistent feature names, and Hierarchical Data Format version 5 (HDF5) storage. Methods: We evaluated Image Biomarker Standardisation Initiative (IBSI) compliance, CPU-CUDA concordance, and end-to-end processing time. Compliance testing included 825 chapter 1 (IBSI-1) tests covering 165 high-consensus features and 323 chapter 2 (IBSI-2) tests with numerical references. Concordance testing included 1,148 scalar pairs and 93 spatial-map pairs. End-to-end processing time was measured five times per input volume of interest (VOI) size. Comparisons included the Medical Image Radiomics Processor (MIRP) and PyRadiomics for 102 shared scalar features and PyRadiomics for 93 shared spatial maps. Results: Both backends passed all 1,148 IBSI tests, and all paired results were concordant. At the largest scalar input, CPU required 76.343 s and hybrid CUDA 81.915 s; CPU was 4.7 times faster than MIRP and 190.7 times faster than PyRadiomics. At the largest spatial input completed by both backends, hybrid CUDA reduced processing time by 68.6% relative to CPU (79.280 versus 252.791 s). At PyRadiomics' largest completed spatial input, hybrid CUDA was 84.9 times faster. Conclusions: Flash-Radiomics unified standardized scalar extraction, spatial mapping, concordant CPU-CUDA results, and HDF5 storage. CPU processing time was similar or shorter for scalar extraction, whereas hybrid CUDA was faster for spatial mapping under the tested conditions.

q-bio.OT↗
arXiv2026

EPI-KAN: A Method For Estimating and Forecasting Time-Dependent COVID-19 Parameters

We introduce EPI-KAN, a novel method for estimating COVID-19 time-varying parameters. EPI-KAN uses historical epidemiological data, Physics-Informed Neural Network (PINN), and the novel Kolmogorov-Arnold Network (KAN). The method harnesses the novel Kolmogorov-Arnold Network (KAN), which is a type of artificial neural network. For the KAN in this paper, we learn activation functions that are represented using Fourier series, hence we abbreviate as KAN-F. In this study, we estimate parameters in the context of an SIRD compartmental differential equations. The time-dependent parameters are the transmission rate $β(t)$, recovery rate $γ(t)$, and mortality rate $μ(t)$. We define three KAN-F functions $\widehatβ$, $\widehatγ$, $\widehatμ$ that model the true parameters $β(t)$, $γ(t)$, $μ(t)$, respectively. We test two model architectures for the KAN-F: the first has 8 input variables consisting of $S$, $I$, $R$, $D$, and their numerical gradients at any time $t$, while the second has 4 input variables excluding the numerical gradients. The objective loss function that has to be minimized is subject to Physics-Informed Neural Network (PINN). Using historical data of COVID-19 from three South-East Asian countries: Indonesia, Singapore, and Malaysia, we are able to estimate $β(t)$, $γ(t)$, and $μ(t)$ on each country with decent accuracy and efficiency. The time period of choice coincides with the period where SARS-CoV-2 Delta variant (B.1.617.2) was dominant. In addition to estimating the rates during the training period, we also predict transmission rates over 30 days during forecast period. We found that the output of KAN-F over the forecast period can give good predictions if we scale the output by a factor of 17\% for Indonesia and 30\% for Singapore and Malaysia.

q-bio.OT↗