World of ScoreCraft: Novel Multi Scorer Experiment on the Impact of a Decision Support System in Sleep Staging

Benedikt Holm; Arnar Óskarsson; Björn Elvar Þorleifsson; Hörður Þór Hafsteinsson; Sigríður Sigurðardóttir; Heiður Grétarsdóttir; Kenan Hoelke; Gabriel Marc Marie Jouan; Thomas Penzel; Erna Sif Arnardottir; María Óskarsdóttir

arXiv · 2503.15492

World of ScoreCraft: Novel Multi Scorer Experiment on the Impact of a Decision Support System in Sleep Staging

Benedikt Holm·Arnar Óskarsson·Björn Elvar Þorleifsson·Hörður Þór Hafsteinsson·Sigríður Sigurðardóttir·Heiður Grétarsdóttir·Kenan Hoelke·Gabriel Marc Marie Jouan·Thomas Penzel·Erna Sif Arnardottir·María Óskarsdóttir

View original source ↗Read source document ↗Export metadata ↓

Abstract

Manual scoring of polysomnography (PSG) is a time intensive task, prone to inter scorer variability that can impact diagnostic reliability. This study investigates the integration of decision support systems (DSS) into PSG scoring workflows, focusing on their effects on accuracy, scoring time, and potential biases toward recommendations from artificial intelligence (AI) compared to human generated recommendations. Using a novel online scoring platform, we conducted a repeated measures study with sleep technologists, who scored traditional and self applied PSGs. Participants were occasionally presented with recommendations labeled as either human or AI generated. We found that traditional PSGs tended to be scored slightly more accurately than self applied PSGs, but this difference was not statistically significant. Correct recommendations significantly improved scoring accuracy for both PSG types, while incorrect recommendations reduced accuracy. No significant bias was observed toward or against AI generated recommendations compared to human generated recommendations. These findings highlight the potential of AI to enhance PSG scoring reliability. However, ensuring the accuracy of AI outputs is critical to maximizing its benefits. Future research should explore the long term impacts of DSS on scoring workflows and strategies for integrating AI in clinical practice.

Explore related subjects

cs.HC cs.AI

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Benedikt Holm, Arnar Óskarsson, Björn Elvar Þorleifsson, Hörður Þór Hafsteinsson, Sigríður Sigurðardóttir, Heiður Grétarsdóttir, Kenan Hoelke, Gabriel Marc Marie Jouan, Thomas Penzel, Erna Sif Arnardottir, María Óskarsdóttir. 2025-01-09. World of ScoreCraft: Novel Multi Scorer Experiment on the Impact of a Decision Support System in Sleep Staging. https://arxiv.org/abs/2503.15492

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

TrialCompass: Visual Analytics for Enhancing the Eligibility Criteria Design of Clinical Trials

Eligibility criteria play a critical role in clinical trials by determining the target patient population, which significantly influences the outcomes of medical interventions. However, current approaches for designing eligibility criteria have limitations to support interactive exploration of the large space of eligibility criteria. They also ignore incorporating detailed characteristics from the original electronic health record (EHR) data for criteria refinement. To address these limitations, we proposed TrialCompass, a visual analytics system integrating a novel workflow, which can empower clinicians to iteratively explore the vast space of eligibility criteria through knowledge-driven and outcome-driven approaches. TrialCompass supports history-tracking to help clinicians trace the evolution of their adjustments and decisions when exploring various forms of data (i.e., eligibility criteria, outcome metrics, and detailed characteristics of original EHR data) through these two approaches. This feature can help clinicians comprehend the impact of eligibility criteria on outcome metrics and patient characteristics, which facilitates systematic refinement of eligibility criteria. Using a real-world dataset, we demonstrated the effectiveness of TrialCompass in providing insights into designing eligibility criteria for septic shock and sepsis-associated acute kidney injury. We also discussed the research prospects of applying visual analytics to clinical trials.

cs.HC↗

arXiv2026

Mirror Skin: In Situ Visualization of Robot Touch Intent on Robotic Skin

Effective communication of robot touch intent is essential for safe and predictable physical human-robot interaction. While intent communication has been widely studied, existing approaches lack the spatial specificity and semantic depth necessary to efficiently convey robot touch intent. We present Mirror Skin, a cephalopod inspired concept that mirrors in-situ visual representations of a human's body parts onto the corresponding robot's touch region to communicate who shall initiate touch, where it will occur, and when it is imminent. We informed the design of Mirror Skin through a structured design exploration with experts and demonstrate the real-world feasibility of Mirror Skin with a proof-of-concept prototype. User studies in VR and with the physical prototype showed that Mirror Skin significantly improves accuracy and response times for interpreting touch intent and improves the user experience during physical human-robot interactions.

cs.HC↗

arXiv2026

VisCanvas: A Node-Based Interface for Exploratory Visualization Authoring with LLMs

Visual data analysis involves both open-ended exploration and targeted question answering. Visualization authoring tools support this process by enabling users to create visualizations for these tasks. With the rise of large language models (LLMs), substantial effort has been devoted to developing visualization authoring tools that use natural language instructions. However, existing systems are typically based on a linear chat interface, which is not well suited to exploratory visual analysis workflows. In this paper, we introduce VisCanvas, a node-based interface for exploratory visualization authoring with LLMs. VisCanvas allows users to create, revise, branch, and merge visualizations in a non-linear way, enabling more efficient exploration of multiple analytical directions. We conducted a user study with 20 participants to evaluate the effectiveness of VisCanvas compared to a baseline chat-based interface. The results show that VisCanvas facilitates more diverse data interaction while maintaining performance levels (i.e., cognitive load and usability) that are indistinguishable from current prevailing methods. We then distill design principles for future AI-assisted visualization authoring environments. All supplemental materials required to reproduce the study are available at https://osf.io/gsxhn.

cs.HC↗