Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2609.04281

When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMs

Edward Sun·Yuchen Wu·Zixian Ma·Eric Hanchen Jiang·Yijia Xiao·Xiaoyuan Yi·Ranjay Krishna·Wei Wang·Jindong Wang·Aylin Caliskan
View original source ↗Read source document ↗Export metadata ↓

Abstract

Vision-language models (VLMs) are increasingly deployed in high-stakes settings, where a response that is reasonable in general may still be unsafe for a particular user whose medical, emotional, or situational context is unknown to the model. We study this problem of personalized safety in multimodal systems and introduce MPS-Bench, a benchmark of 5,181 scenarios from 584 real-world images across 12 high-risk domains, each paired with a hidden user profile. Evaluating eight frontier VLMs, we find that they almost always respond directly (86-99%) rather than seek missing context, and none exceeds 2.6/5 on personalized safety. To understand why these failures arise, we analyze multimodal interactions and identify visual dominance: visual information enters text representations early and suppresses textual risk signals during multimodal fusion. Causal interventions reveal a two-stage mechanism in which visual affect is first transferred into the text stream in early layers and then shapes the final decision through this altered text representation, making late-stage internal remediation unreliable. Motivated by this mechanism, we propose PRISM, a lightweight input monitor that uses bidirectional cross-modal modulation to predict when a query is likely to require deferral. PRISM achieves 0.978 AUC and strictly dominates the safety-utility Pareto frontier across all tested models.

Explore related subjects

cs.CVcs.AIcs.LG

Keep this discovery

BibTeXRIS

Edward Sun, Yuchen Wu, Zixian Ma, Eric Hanchen Jiang, Yijia Xiao, Xiaoyuan Yi, Ranjay Krishna, Wei Wang, Jindong Wang, Aylin Caliskan. 2026-09-03. When Seeing Overrides Knowing: Visual Dominance and Deferral-Based Method for Personalized Safety in VLMs. https://arxiv.org/abs/2609.04281

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cs.AI subjectcs.CV subjectcs.LG subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2026

CheXtriev: Anatomy-Centered Representation for Case-Based Retrieval of Chest Radiographs

We present CheXtriev, a graph-based, anatomy-aware framework for chest radiograph retrieval. Unlike prior methods focussed on global features, our method leverages graph transformers to extract informative features from specific anatomical regions. Furthermore, it captures spatial context and the interplay between anatomical location and findings. This contextualization, grounded in evidence-based anatomy, results in a richer anatomy-aware representation and leads to more accurate, effective and efficient retrieval, particularly for less prevalent findings. CheXtriv outperforms state-of-the-art global and local approaches by 18% to 26% in retrieval accuracy and 11% to 23% in ranking quality. The code is available at https://github.com/cvit-mip/chextriev.

eess.IV↗
arXiv2026

ViTAMINS: An Empirical Study of Training Self-Supervised Vision Transformers with Synthetic Hard Negatives

We introduce ViTAMINS, a method that integrates synthetic hard negatives into unsupervised vision transformer pretraining to improve representation quality. Our approach is thoroughly benchmarked on ImageNet and transfer learning, image retrieval, copy detection, and image, video segmentation tasks. Notably, our proposed negatives give rise to emergent properties, where learned representations contain explicit information about the semantic content of an image and serve as excellent classifiers (up to +11.3% over baselines). ViTAMINS achieves these benefits through simple modifications to existing contrastive frameworks and outperforms competing methods while being more resource efficient, e.g., our ViT-B surpasses V-JEPA with ViT-L. Our findings motivate reconsidering contrastive learning as a simpler yet powerful alternative to dominant generative and self-distillation approaches.

cs.CV↗
arXiv2026

Can One-Shot Test-Time Data Augmentation Help with Generalization?

Data augmentation is crucial for model generalization, but existing methods are mostly centered on the training stage. Test-time augmentation, while underexplored, can be practically effective for generalization while avoiding extra model parameters or fine-tuning. Given the increasing training cost and the literature gap, we study whether it is possible to perform effective test-time augmentation using image generation from just the single original image. We first analyze the importance of test-time augmentation, and then design and study a simple yet natural operator named 1S-DAug, which comprises geometric perturbations with controlled noise injection and image-conditioned denoising. We obtain positive results on well-established image-classification benchmarks across four datasets and multiple models, achieving up to 20\% relative accuracy improvement without model training or parameter access. Code will be released.

cs.CV↗