Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2609.00524

Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications

Satwik Ram Kodandaram·Monalika Padma Reddy·Xiaojun Bi·Jiawei Zhou·I. V. Ramakrishnan·Vikas Ashok
View original source ↗Read source document ↗Export metadata ↓

Abstract

Computer-use agents are emerging as a paradigm for agentic human-AI interaction, combining language reasoning with multi-modal interface grounding to operate GUIs. Yet their effectiveness for blind screen-reader users in real-world desktop workflows remains unclear. We present a three-week diary study with 8 blind users using OLLA, a screen-reader-accessible CUA prototype, collecting 1,258 commands across 12 applications with screenshots, UI trees, model responses, and action traces. We evaluate GPT-5 during deployment and re-execute the same commands with four additional models. GPT-5 achieved the highest success rate at 52.5%. Trace analysis reveals grounding, planning, constraint-tracking, and termination failures, while interviews reveal beyond-automation needs.

Explore related subjects

cs.HCcs.AI

Keep this discovery

BibTeXRIS

Satwik Ram Kodandaram, Monalika Padma Reddy, Xiaojun Bi, Jiawei Zhou, I. V. Ramakrishnan, Vikas Ashok. 2026-09-01. Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications. https://arxiv.org/abs/2609.00524

Cite the original work for its findings. Save a collection to share your selection of sources.

Discover connections

cs.AI subjectcs.HC subject

Connections use source metadata and explicit phrase matches, not verified experimental comparisons.

KEEP EXPLORING

Related papers

arXiv2026

TSExplorer: An interactive data annotation and exploration tool for time-series data

We present TSExplorer, a cross-platform tool for interactive annotation and exploration of time-series data. The tool enables users to inspect high-dimensional datasets through multiple complementary 2D visualizations derived from high-dimensional feature representations. TSExplorer is designed as a general-purpose research tool supporting a wide range of workflows, including exploratory data analysis, annotation of unlabeled or partially-labeled datasets, comparison of feature representations, and post-hoc inspection and refinement of existing labels with interactive visual feedback.

cs.HC↗
arXiv2026

Experts Disagree on How to Fight AI Disinformation, but Agree That Health and Politics Need Different Solutions

When 54 international experts assessed AI-generated disinformation threats, they revealed a surprising pattern: while video deepfakes received the highest average threat ratings in the political domain (M = 6.31/7), the pattern differed in the health domain, where AI-generated text received the highest average rating (M = 5.80). Experts also diverge on what to do: government regulation drew both the most "most effective" (30%) and the most "least effective" (15%) votes, though rating distributions were contested rather than polarized, indicating disagreement over priorities rather than over efficacy. These findings offer an initial expert map of an AI-disinformation landscape that is still rapidly forming.

cs.CY↗
arXiv2026

Unsupervised Partner Design Enables Robust Ad-hoc Teamwork

We introduce Unsupervised Partner Design (UPD), a population-free multi-agent reinforcement learning method for robust ad-hoc teamwork. UPD generates training partners on-the-fly and selects them adaptively based on a learnability criterion, removing the need for pre-trained partner populations or manual parameter tuning. We show that this simple mechanism enables effective partner diversity and can be extended to joint partner-environment selection when a procedural level generator is available. Across Level-Based Foraging, Overcooked-AI, and the Overcooked Generalisation Challenge, UPD consistently achieves strong performance compared to both population-based and population-free baselines. In a human-AI user study, agents trained with UPD achieve higher returns and are rated as more adaptive, more human-like, and less frustrating than all evaluated baseline methods.

cs.LG↗