Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.32883

When Should I Trust This? Designing and Evaluating Factuality Indicators for LLM-based Chatbots

Greta Warren·Jessica Heal·Sila Eser·Denise M. Leto·Christine M. Lehane·Isabelle Augenstein
View original source ↗Read source document ↗Export metadata ↓

Abstract

Generative AI systems increasingly mediate how people seek information, yet communicating AI factuality to laypeople remains challenging. Through a mixed-methods investigation involving interviews (N=8), a survey (N=1,000), and an experiment (N=5,151), we examine how people evaluate AI-generated information for gardening, mathematical, and medical queries, and how factuality indicators (confidence and sources) shape perceptions. Our interviews and survey show people selectively verify AI responses using external and AI-provided evidence, moderated by contextual factors. Although confidence-indicators affected trust and accuracy perceptions, people hold diverse, ambiguous interpretations of AI confidence. Sources were viewed as crucial, yet users raised concerns about source reliability and verification effort. Our experiment showed that confidence and source-based indicators influenced initial perceptions, but effects disappeared once participants learned the response's correctness. Medical queries elicited skepticism regardless of indicator. Findings motivate helping users interpret model confidence and integrating source information into the interface to support calibrated trust in AI-supported information-seeking.

Explore related subjects

cs.HC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Greta Warren, Jessica Heal, Sila Eser, Denise M. Leto, Christine M. Lehane, Isabelle Augenstein. 2026-10-23. When Should I Trust This? Designing and Evaluating Factuality Indicators for LLM-based Chatbots. https://arxiv.org/abs/2610.32883

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Tool and Trainer? Experimental Evidence on Human Learning from Predictive AI in Cancer Diagnosis

Artificial intelligence (AI) is increasingly deployed to aid professional judgment, yet its implications for the development of human expertise remain unclear. We examine how the availability of AI assistance during training and task execution shapes novices' individual and collective judgment in the context of cancer diagnosis. Across two field experiments at a large teaching hospital, 576 medical novices learned to diagnose lung cancer from CT scans. Study 1 used a 2 * 2 factorial design that randomized AI assistance during training, task execution, both, or neither. Novices who received AI assistance during training achieved higher diagnostic accuracy in both AI-assisted and unaided task execution, with a larger improvement in the former. However, training-phase assistance was associated with more common errors than task-execution-phase assistance, suggesting that while the trainer effect improves individual accuracy, it limits the benefits of aggregating judgments. Study 2 shows that AI assistance during earlier task execution improved subsequent unaided performance and reduced common errors in the unaided phase of task execution, thus revealing a surprisingly beneficial "tool-as-trainer" effect. These findings suggest that, contingent on the right deployment strategy, AI assistance can support individual expertise development and improve collective judgment by reducing common errors across individuals.

cs.HC↗
arXiv2026

KnowledgeTrail: Understanding Patterns of Exploratory Sensemaking and Serendipitous Discovery in Generative Timelines

Generative AI is reshaping how interactive systems support the discovery and understanding of knowledge, such as how events and ideas developed over time. Yet timelines, one of the most fundamental tools for enabling this kind of temporal sensemaking, remain largely static, limiting user agency and curiosity-driven exploration. Moreover, little is known about how people may explore and make sense of information if the structure of a timeline itself can evolve through interaction. To investigate this question, we designed KnowledgeTrail, a generative timeline that grows and adapts to users' questions, expanding and contracting with their evolving curiosity. Through an exploratory study, we examined how people engage with KnowledgeTrail and identified several patterns of curiosity-driven exploration, serendipitous discovery, and relational sensemaking that differ from conversational AI interfaces. Based on these analyses, our work contributes design implications for future generative systems that support dynamic, user-driven tracing of ideas, events, and their relationships.

cs.HC↗
arXiv2026

PAIR-SAFE: A Paired-Agent Approach for Runtime Auditing and Refining AI-Mediated Mental Health Support

LLMs are increasingly used for mental health support, yet they can produce responses that are overly directive, inconsistent, or clinically misaligned. Most risk mitigation approaches rely on implicit alignment through training or prompting, offering limited runtime accountability. We introduce PAIR-SAFE, a paired-agent framework that integrates a Responder agent with a Judge agent grounded in the clinically validated Motivational Interviewing Treatment Integrity (MITI-4). The Judge audits each response and issues \allow{} or \revise{} decisions to guide real-time refinement. We evaluate interactions using a support-seeker simulator derived from human-annotated data. The Judge-supervised interactions show significant improvements in key MITI dimensions, including Partnership, Seek Collaboration, and overall Relational quality. Our quantitative findings are corroborated by expert evaluation, highlighting nuances of runtime supervision. Our study reveals that paired-agent auditing can provide clinically grounded, transparent oversight for AI-assisted mental health support.

cs.HC↗