Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design

Jingyi Xie; Rui Yu; He Zhang; Sooyeon Lee; Syed Masum Billah; John M. Carroll

Home/Archive/arXiv

arXiv · 2407.08882

Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design

Jingyi Xie·Rui Yu·He Zhang·Sooyeon Lee·Syed Masum Billah·John M. Carroll

View original source ↗Read source document ↗Export metadata ↓

Abstract

People with visual impairments perceive their environment non-visually and often use AI-powered assistive tools to obtain textual descriptions of visual information. Recent large vision-language model-based AI-powered tools like Be My AI are more capable of understanding users' inquiries in natural language and describing the scene in audible text; however, the extent to which these tools are useful to visually impaired users is currently understudied. This paper aims to fill this gap. Our study with 14 visually impaired users reveals that they are adapting these tools organically -- not only can these tools facilitate complex interactions in household, spatial, and social contexts, but they also act as an extension of users' cognition, as if the cognition were distributed in the visual information. We also found that although the tools are currently not goal-oriented, users accommodate this limitation and embrace the tools' capabilities for broader use. These findings enable us to envision design implications for creating more goal-oriented, real-time processing, and reliable AI-powered assistive technology.

Explore related subjects

cs.HC

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Jingyi Xie, Rui Yu, He Zhang, Sooyeon Lee, Syed Masum Billah, John M. Carroll. 2024-07-11. Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Design. https://arxiv.org/abs/2407.08882

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

How Do Users Negotiate Harmful Value Conflicts with AI Companions? A Study with Minion, a Technology Probe for In-Situ Human-AI Conflict Response

AI companions increasingly sustain long-term, emotionally engaging relationships but can also make discriminatory remarks or exert control, leaving users to manage harmful conflicts. We analyze 146 posts describing harmful value conflicts with AI companions, then use Minion, a technology probe offering response suggestions ranging from persuasion to boundary setting, to study how 22 users negotiate scenario-based conflicts over one week. We found that participants combined softer and harder strategies. Conflicts involving the values of Universalism and Tradition were especially difficult to negotiate, particularly when reinforced by AI personas or platform constraints. We argue that these conflicts entail asymmetric responsibility: users draw on an interpersonal repertoire that AI companions cannot reciprocate, making repair unilateral safety work. Drawing on interpersonal conflict and communication theory, we identify when user-side support is appropriate and argue that certain harms are not users' responsibility to negotiate and instead require platform-level safeguards.

cs.HC↗

arXiv2026

Conversational DNA: A Visual Language and Interactive Atlas of Human and AI Dialogue

What makes a conversation hold together when its participants speak across one another? Topic maps offer one view, but they leave the relationships between contributions difficult to inspect. We present Conversational DNA, a visual language and interactive atlas for exploring human and AI dialogue. Speaker strands preserve participation, communicative bases mark moves, and directed pairings connect responses to their targets. Adjustable helix geometry makes speaker switching, response distance, and contribution length visible. Across eight corpora containing 1.57 million source records, the atlas maps 151,489 indexed episodes and connects cohort comparison to source transcripts, local structural alignment, and recorded reply alternatives. On 189 held-out Molweni motif queries, adding target correspondence improves precision@5 from 58.8% to 77.2% for exact annotated structure. Case readings illustrate interleaved participation, delayed responses, and the influence of annotation coverage on apparent collection differences. The system supports a view of conversation as jointly organized activity, with visual patterns serving as starting points for examining evidence rather than substitutes for interpretation.

cs.HC↗

arXiv2026

Human- vs. AI-generated tests: dimensionality and information accuracy in latent trait evaluation

Artificial Intelligence (AI) and large language models (LLMs) are increasingly used in social and psychological research. Among potential applications, LLMs can be used to generate, customise, or adapt measurement instruments. This study presents a preliminary investigation of AI-generated questionnaires by comparing two ChatGPT-based adaptations of the Body Awareness Questionnaire (BAQ) with the validated human-developed version. The AI instruments were designed with different levels of explicitness in content and instructions on construct facets, and their psychometric properties were assessed using a Bayesian Graded Response Model. Results show that although surface wording between AI and original items was similar, differences emerged in dimensionality and in the distribution of item and test information across latent traits. These findings illustrate the importance of applying statistical measures of accuracy to ensure the validity and interpretability of AI-driven tools.

cs.HC↗