Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2607.15565

Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models

Rakshanda Hassan Abhinandan·John Galeotti·Deva Ramanan·Gautam Rajendrakumar Gare
View original source ↗Read source document ↗Export metadata ↓

Abstract

Where should the question go in a vision-language model (VLM) prompt: before the image or after it? Intuition says before: knowing what is asked should tell the model where to look. Yet across visual question answering benchmarks, question-first prompting consistently underperforms the image-first ordering recommended for frontier VLMs, a phenomenon we term the question-first paradox. We trace this paradox to a conflict between two stages of VLM computation. Logit-lens and attention probes show that question-first prompting steers perception, shifting image patch representations toward question-relevant concepts. But downstream, stranded behind hundreds of image tokens, the question is barely attended by the answer token, which instead commits to image-driven, often wrong answers. Causal attention knockout confirms that the answer reads the question only when it follows the image. This diagnosis yields a training-free fix: question echoing, restating the question on both sides of the image so one copy steers perception while the other is available at answer time. A similar division of labor appears in a fifty-year-old finding on human 'adjunct questions', where repeating a question before and after a passage improves comprehension. Echoing the image as well brings further gains by restoring the whole-image view otherwise lost by a causal decoder. The paradox holds across five open VLMs, costing up to 17.5 group-accuracy points. Echoed prompts recover most of the gap and, on NaturalBench and Winoground, surpass the best single-pass ordering by up to 19 group-accuracy points on Winoground, with no training, fine-tuning, or architecture change. The paradox reveals a tension between steering what a model sees and preserving access to what it was asked; echoing resolves this through prompt design. Project Page: https://rakshanda-cmu.github.io/ask-twice-look-twice/

Explore related subjects

cs.CVcs.AIcs.LGeess.IV

Keep this discovery

BibTeXRIS

Rakshanda Hassan Abhinandan, John Galeotti, Deva Ramanan, Gautam Rajendrakumar Gare. 2026-09-03. Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models. https://arxiv.org/abs/2607.15565

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cs.AI subjectcs.CV subjectcs.LG subjecteess.IV subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2026

CheXtriev: Anatomy-Centered Representation for Case-Based Retrieval of Chest Radiographs

We present CheXtriev, a graph-based, anatomy-aware framework for chest radiograph retrieval. Unlike prior methods focussed on global features, our method leverages graph transformers to extract informative features from specific anatomical regions. Furthermore, it captures spatial context and the interplay between anatomical location and findings. This contextualization, grounded in evidence-based anatomy, results in a richer anatomy-aware representation and leads to more accurate, effective and efficient retrieval, particularly for less prevalent findings. CheXtriv outperforms state-of-the-art global and local approaches by 18% to 26% in retrieval accuracy and 11% to 23% in ranking quality. The code is available at https://github.com/cvit-mip/chextriev.

eess.IV↗
arXiv2026

ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives

We introduce ViTAMINS, a method that integrates synthetic hard negatives into unsupervised vision transformer pretraining to improve representation quality. Our approach is thoroughly benchmarked on ImageNet and transfer learning, image retrieval, copy detection, and image, video segmentation tasks. Notably, our proposed negatives give rise to emergent properties, where learned representations contain explicit information about the semantic content of an image and serve as excellent classifiers (up to +11.3% over baselines). ViTAMINS achieves these benefits through simple modifications to existing contrastive frameworks and outperforms competing methods while being more resource efficient, e.g., our ViT-B surpasses V-JEPA with ViT-L. Our findings motivate reconsidering contrastive learning as a simpler yet powerful alternative to dominant generative and self-distillation approaches.

cs.CV↗
arXiv2026

Can One-Shot Test-Time Data Augmentation Help with Generalization?

Data augmentation is crucial for model generalization, but existing methods are mostly centered on the training stage. Test-time augmentation, while underexplored, can be practically effective for generalization while avoiding extra model parameters or fine-tuning. Given the increasing training cost and the literature gap, we study whether it is possible to perform effective test-time augmentation using image generation from just the single original image. We first analyze the importance of test-time augmentation, and then design and study a simple yet natural operator named 1S-DAug, which comprises geometric perturbations with controlled noise injection and image-conditioned denoising. We obtain positive results on well-established image-classification benchmarks across four datasets and multiple models, achieving up to 20\% relative accuracy improvement without model training or parameter access. Code will be released.

cs.CV↗