Speech recognition assisted by large language models to command software orally -- Application to an augmented and virtual reality web app for immersive molecular graphics

Fabio Cortes Rodriguez; Luciano Abriata

Home/Archive/arXiv

arXiv · 2603.02901

Speech recognition assisted by large language models to command software orally -- Application to an augmented and virtual reality web app for immersive molecular graphics

Fabio Cortes Rodriguez·Luciano Abriata

View original source ↗Read source document ↗Export metadata ↓

Abstract

This project successfully developed, evaluated and integrated a Voice User Interface (VUI) into a web application that we are developing for immersive molecular graphics. Said app provides augmented and virtual reality (AR and VR) environments where users manipulate molecules with their hands, but this means the hands can't be used to control the app through a regular mouse- and keyboard-based GUI. The speech-based VUI system developed here alleviates this problem, making it easy to control the app via natural spoken (or typed) commands. To achieve this VUI we evaluated two distinct Automated Speech Recognition (ASR) systems: Chrome's native Speech API and OpenAI's Whisper v3. While Whisper offered broader browser compatibility, its tendency to "hallucinate" with specialized scientific jargon proved very problematic. Consequently, we selected Chrome's ASR for its stability, speed, and reliability. For translating transcribed speech into software commands, we tested two Large Language Model (LLM)-driven approaches: either generating executable code, or calling predefined functions. The function call method, powered by OpenAI's GPT-4o-mini, was ultimately adopted due to its superior safety, efficiency, and reliability over the more complex and error-prone code-generation approach. The resulting VUI is then based on an integration of Chrome's ASR with our LLM-based function-calling module, enabling users to command the application using natural language as shown in a video linked inside this report. We provide links to live examples demonstrating all the intermediate components, and details on how we crafted the LLM's prompt in order to teach it the function calls as well as ways to clean up the transcribed speech and to explain itself while generating function calls. For best demonstration of the final system, we provide a video example.

Explore related subjects

cs.HC

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Fabio Cortes Rodriguez, Luciano Abriata. 2026-03-03. Speech recognition assisted by large language models to command software orally -- Application to an augmented and virtual reality web app for immersive molecular graphics. https://arxiv.org/abs/2603.02901

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Learning Password Best Practices Through In-Task Instruction

Users often make security- and privacy-relevant decisions without a clear understanding of the rules that govern safe behavior. We introduce pedagogical friction, a design approach that inserts brief, instructional interactions at the moment of action. We evaluate this approach in the context of password creation, a familiar task with clear quality criteria. We conducted a randomized study with 128 participants across four interface conditions that varied the depth and interactivity of guidance. We assessed three outcomes: (1) rule compliance in a subsequent password task without guidance, (2) accuracy on survey questions tied to password rules, and (3) behavior-knowledge alignment, which captures whether participants who correctly followed a rule also recognized it on the survey. Across the guided conditions, participants corrected most rule violations in the follow-up task and showed high behavior-knowledge alignment. Survey results suggested clearer advantages for some rule types, especially symbol related questions. These results position pedagogical friction as a lightweight intervention for security- and privacy-critical interfaces.

cs.HC↗

arXiv2026

Leveling the Playing Field: Temporal Video Segmentation for Individuals with ADHD in Computing Education

Individuals with Attention-Deficit/Hyperactivity Disorder (ADHD) often face significant barriers in computing education. In asynchronous learning environments, instructional videos can impose high extraneous cognitive load, often relying on assumptions about sustained attention and working memory that do not align with ADHD neurocognitive profiles. In this work, we evaluate a post-hoc video processing intervention that segments instructional content into single-instruction chunks followed by fixed-length pauses to reduce cognitive load. In a within-participants controlled study with 17 individuals with ADHD and 10 without, we find that the intervention has an equalizing effect. Although it improved performance for all participants, gains were larger for those with ADHD, reducing their errors and hesitations to levels comparable to those of participants without ADHD under the same intervention. These results align with the goals of Universal Design for Learning (UDL), by showing that cognitively-aligned, post-hoc instructional video modifications can reduce performance disparities across diverse neurocognitive profiles.

cs.HC↗

arXiv2026

Human Driver Temperament and the Safety Impact of a C-V2X Denial-of-Service Flooding Attack in Mixed-Autonomy Traffic

Cooperative and connected automated vehicles (CAVs) rely on Signal Phase and Timing (SPaT) messages to cross signalized intersections; a denial-of-service (DoS) flood that blocks SPaT forces CAVs into a fail-safe mode. Because human-driven vehicles share the intersection, the safety consequence depends not only on the attack and the CAV fail-safe policy, but on how the surrounding human drivers behave. We investigate this human-factors dimension with a coupled OMNeT++/INET (5G NR-V2X) and SUMO microsimulation of a signalized corridor, sweeping CAV market penetration (10-90%), four calibrated driver temperaments (cautious to aggressive) and two standards-based fail-safe policies, a minimal-risk maneuver (MRM) and an adaptive cruise control (ACC) keep-driving fallback, with each attack arm differenced against its policy-matched no-attack baseline. Temperament's effect on the attack is specific and modest rather than a blanket amplification. Aggressive surroundings worsen one metric, the hard-braking a keep-driving fail-safe forces on nearby drivers (p = 0.03), rising from near zero to +8 episodes/1000 veh-s. They appear to dampen rear-end conflicts, but only because the flood clears the queues aggressive drivers build, so the gain is in flow, not safety. On the attack's primary signatures, CAV red-light running and crossing conflicts, temperament has no detectable effect. It instead dominates baseline risk, producing a 13- to 18-fold cautious-to-aggressive gradient far larger than the attack itself, which acts through a channel already congested by CAV adoption. Human driver populations determine how dangerous the intersection is but do not systematically amplify this attack, so fail-safe design cannot assume a cautious test population bounds the risk.

cs.HC↗