Search arXivSearch

arXiv subjects

Xinyan Yu

Publications and source records attributed to Xinyan Yu.

At least 19 recordsLinked to original sources

SkillAligner: Treating Retrieved Skills as Adaptable Drafts at Execution Time

General-purpose skills promise reusable procedural knowledge for language agents, yet semantic relevance does not guarantee execution utility: a retrieved skill may encode assumptions that conflict with the current task, execution environment, or other retrieved skills. We formalize this problem as the skill--execution misfit. To address it, we propose SkillAligner, a training-free execution-time skill adaptation framework that treats retrieved skills as adaptable drafts rather than fixed instructions. Before execution, SkillAligner performs a one-time joint adaptation that specializes useful skill fragments to task requirements, aligns their procedural assumptions with the available execution interface, and composes the resulting guidance by resolving dependencies, conflicts, and redundancy across skills. The adapted content is consolidated into a compact execution guide and reused throughout the subsequent trajectory. Extensive experiments across diverse agent benchmarks and model backbones show that SkillAligner substantially improves task performance over existing skill-use baselines, reduces skill-induced regressions at the instance level, and lowers total inference cost.

cs.LG

AttriMem: Attribution-Guided Process Feedback for Agent Memory Construction

Effective memory is crucial for LLM agents, yet constructing it effectively remains challenging. A memory-construction policy decides what information to extract, store, update, compress, or discard as interactions accumulate. Heuristic memory methods rely on subjective, task-specific rules, which can misalign with downstream objectives and limit cross-task adaptability. RL-based methods, by contrast, learn from task feedback but mainly use outcome- or module-level rewards. These coarse signals indicate task success but cannot identify which intermediate memory contents support the final answer, creating a fine-grained credit-assignment bottleneck. However, constructing such process feedback is prohibitively difficult because intermediate memory decisions lack unique ground-truth targets, while the appropriate credit varies with the agent's uncertain reasoning trajectory and therefore cannot be specified in advance. We propose AttriMem, an attribution-guided process-feedback framework for learning memory-construction policies with RL. AttriMem augments the global outcome reward with local rewards derived from token-level contributions to the final answer. Experiments on long-horizon dialogue question answering show that AttriMem outperforms retrieval-based, heuristic, and RL-based baselines, generalizes across benchmarks and answer models, stabilizes RL optimization.

cs.AI

Speculating the Impacts of Mediated Social Touch Technology

With growing research on haptic interfaces, Mediated Social Touch (MST) technologies offer the potential to record, synthesise, and reproduce (RSR) touch experiences across space and time, enabling, for instance, a hug from afar and from the past. Although much of the existing research highlights the direct benefits of these systems, such as reducing loneliness and providing emotional support, little attention has been paid to their broader sociotechnical impacts. To address this gap, we used the Future Ripples method to speculate on possible effects of MST. We conducted three workshops with 24 participants, including potential users, domain experts, and haptics researchers. Throughout these sessions, participants collectively envisioned possible future scenarios, alongside opportunities and threats, and proposed actionable responses. Our qualitative analysis organised these insights into four themes and three distinctive challenges. These findings offer haptics researchers intervention points across the RSR pipeline to inform MST design, alongside methodological insights from applying Future Ripples to MST technology.

cs.HC

Towards Steering without Sacrifice: Principled Training of Steering Vectors for Prompt-only Interventions

Recently, steering vectors (SVs) have emerged as an effective and lightweight approach to steer behaviors of large language models (LLMs), among which fine-tuned SVs are more effective than optimization-free ones. However, current approaches to fine-tuned SVs suffer from two limitations. First, they require careful selection of steering factors on a per-SV basis to balance steering effectiveness and generation quality at inference time. Second, they operate as full-sequence SVs (FSSVs), which can sacrifice generation quality regardless of factor selection due to excessive intervention on the model generation process. To address the first limitation, we propose joint training of steering factors and directions, such that post-hoc factor selection is no longer required. Using neural network scaling theory, we find that moderately large initialization sizes and learning rates for steering factors are essential for stability and efficiency of joint training. To tackle the second limitation, we draw inspiration from representation fine-tuning and introduce Prompt-only SV (PrOSV), an SV that intervenes only on a few prompt tokens. Our empirical results show that PrOSV outperforms traditional FSSVs on AxBench when using our joint training scheme. We also find that PrOSV achieves a better tradeoff between general model utility and adversarial robustness than FSSV.

cs.LG

Animated Public Furniture as an Interaction Mediator: Engaging Passersby In-the-Wild with Robotic Benches

Urban HCI investigates how digital technologies shape human behaviour within the social, spatial, temporal dynamics of public space. Meanwhile, robotic furniture research demonstrates how the purposeful animation of mundane utilitarian elements can influence human behaviour in everyday contexts. Taken together, these strands highlight an untapped opportunity to investigate how animated public furniture could mediate social interaction in urban environments. In this paper, we present the design process and in-the-wild study of mobile robotic benches that reconfigure with a semi-outdoor public space. Our findings show that the gestural performance of the benches manifested three affordances perceived by passersby, they activated engagement as robots, redistributed engagement as spatial elements, and settled engagement as infrastructure. We proposed an Affordance Transition Model (ATM) describing how robotic furniture could proactively facilitate transition between these affordances to engage passersby. Our study bridges robotic furniture and urban HCI to activate human experience with the built environment purposefully.

cs.HC

Fostering Design-Policy Collaboration through Contestation: An Adversarial Futuring Method

Emerging technologies introduce sociotechnical tensions that call for closer collaboration between technology design and policy. In this work, we introduce Design-Policy Adversarial Futuring, a scenario-based workshop method that supports design-policy engagement by structuring contestation between design and policy perspectives. We report on a workshop conducted in the autonomous mobility domain with 12 HCI researchers, used to explore and demonstrate the method in practice. The workshop illustrates how the adversarial futuring method can surface shifting harms, translate policy abstractions into situated use, and legitimise extreme ideas while maintaining grounded policy reasoning. This work contributes a reusable, exploratory method for supporting HCI-policy collaboration through contestation, which can be adapted across emerging technological domains.

cs.HC

Envisioning Audio Augmented Reality in Everyday Life

While visual augmentation dominates the augmented reality landscape, devices like Meta Ray-Ban audio smart glasses signal growing industry movement toward audio augmented reality (AAR). Hearing is a primary channel for sensing context, anticipating change, and navigating social space, yet AAR's everyday potential remains underexplored. We address this gap through a collaborative autoethnography (N=5, authoring) and an online survey (N=74). We identify ten roles for AAR, grouped into three categories: task- and utility-oriented, emotional and social, and perceptual collaborator. These roles are further layered with a rhythmic and embodied collaborator framing, mapping them onto micro-, meso-, and macro-rhythms of everyday life. Our analysis surfaces nuanced tensions, such as blocking distractions without erasing social presence, highlighting the need for context-aware design. This paper contributes a foundational and forward-looking framework for AAR in everyday life, providing design groundwork for systems attuned to daily routines, sensory engagement, and social expectations.

cs.HC

Feel the Presence: The Effects of Haptic Sensation on VR-Based Human-Robot Interaction

Virtual reality (VR) has been increasingly utilised as a simulation tool for human-robot interaction (HRI) studies due to its ability to facilitate fast and flexible prototyping. Despite efforts to achieve high validity in VR studies, haptic sensation, an essential sensory modality for perception and a critical factor in enhancing VR realism, is often absent from these experiments. Studying an interactive robot help-seeking scenario, we used a VR simulation with haptic gloves that provide highly realistic tactile and force feedback to examine the effects of haptic sensation on VR-based HRI. We compared participants' sense of presence and their assessments of the robot to a traditional setup using hand controllers. Our results indicate that haptic sensation enhanced participants' social and self-presence in VR and fostered more diverse and natural bodily engagement. Additionally, haptic sensations significantly influenced participants' affective-related perceptions of the robot. Our study provides insights to guide HRI researchers in building VR-based simulations that better align with their study contexts and objectives.

cs.HC

The UnScripted Trip: Fostering Policy Discussion on Future Human-Vehicle Collaboration in Autonomous Driving Through Design-Oriented Methods

The rapid advancement of autonomous vehicle (AV) technologies is fundamentally reshaping paradigms of human-vehicle collaboration, raising not only an urgent need for innovative design solutions but also for policies that address corresponding broader tensions in society. To bridge the gap between HCI research and policy making, this workshop will bring together researchers and practitioners in the automotive community to explore AV policy directions through collaborative speculation on the future of AVs. We designed The UnScripted Trip, a card game rooted in fictional narratives of autonomous mobility, to surface tensions around human-vehicle collaboration in future AV scenarios and to provoke critical reflections on design solutions and policy directions. Our goal is to provide an engaging, participatory space and method for automotive researchers, designers, and industry practitioners to collectively explore and shape the future of human-vehicle collaboration and its policy implications.

cs.HC

Towards Adaptive External Communication in Autonomous Vehicles: A Conceptual Design Framework

External Human-Machine Interfaces (eHMIs) are key to facilitating interaction between autonomous vehicles and external road actors, yet most remain reactive and do not account for scalability and inclusivity. This paper introduces a conceptual design framework for adaptive eHMIs-interfaces that dynamically adjust communication as road actors vary and context shifts. Using the cyber-physical system as a structuring lens, the framework comprises three layers: Input (what the system detects), Processing (how the system decides), and Output (how the system communicates). Developed through theory-led abstraction and expert discussion, the framework helps researchers and designers think systematically about adaptive eHMIs and provides a structured tool to design, analyse, and assess adaptive communication strategies. We show how such systems may resolve longstanding limitations in eHMI research while raising new ethical and technical considerations.

cs.HC

Understanding Pedestrian Gesture Misrecognition: Insights from Vision-Language Model Reasoning

Pedestrian gestures play an important role in traffic communication, particularly in interactions with autonomous vehicles (AVs), yet their subtle, ambiguous, and context-dependent nature poses persistent challenges for machine interpretation. This study investigates these challenges by using GPT-4V, a vision-language model, not as a performance benchmark but as a diagnostic tool to reveal patterns and causes of gesture misrecognition. We analysed a public dataset of pedestrian-vehicle interactions, combining manual video review with thematic analysis of the model's qualitative reasoning. This dual approach surfaced recurring factors influencing misrecognition, including gesture visibility, pedestrian behaviour, interaction context, and environmental conditions. The findings suggest practical considerations for gesture design, including the value of salience and contextual redundancy, and highlight opportunities to improve AV recognition systems through richer context modelling and uncertainty-aware interpretations. While centred on AV-pedestrian interaction, the method and insights are applicable to other domains where machines interpret human gestures, such as wearable AR and assistive technologies.

cs.HC

Uncertainty on Display: The Effects of Communicating Confidence Cues in Autonomous Vehicle-Pedestrian Interactions

Uncertainty is an inherent aspect of autonomous vehicle (AV) decision-making, yet it is rarely communicated to pedestrians, which hinders transparency. This study investigates how AV uncertainty can be conveyed through two approaches: explicit communication (confidence percentage displays) and implicit communication (vehicle motion cues), across different confidence levels (high and low). Through a within-subject VR experiment (N=26), we evaluated these approaches in a crossing scenario, assessing interface qualities (visibility and intuitiveness), how well the information conveyed the vehicle's level of confidence, and their impact on participants' perceived safety, trust, and user experience. Our results show that explicit communication is more effective and preferred for conveying uncertainty, enhancing safety, trust, and user experience. Conversely, implicit communication introduces ambiguity, especially when AV confidence is low. This research provides empirical insights into how uncertainty communication shapes pedestrian interpretation of AV behaviour and offer design guidance for external interfaces that integrate uncertainty as a communicative element.

cs.HC

Animal Interaction with Autonomous Mobility Systems: Designing for Multi-Species Coexistence

Autonomous mobility systems increasingly operate in environments shared with animals, from urban pets to wildlife. However, their design has largely focused on human interaction, with limited understanding of how non-human species perceive, respond to, or are affected by these systems. Motivated by research in Animal-Computer Interaction (ACI) and more-than-human design, this study investigates animal interactions with autonomous mobility through a multi-method approach combining a scoping review (45 articles), online ethnography (39 YouTube videos and 11 Reddit discussions), and expert interviews (8 participants). Our analysis surfaces five key areas of concern: Physical Impact (e.g., collisions, failures to detect), Behavioural Effects (e.g., avoidance, stress), Accessibility Concerns (particularly for service animals), Ethics and Regulations, and Urban Disturbance. We conclude with design and policy directions aimed at supporting multispecies coexistence in the age of autonomous systems. This work underscores the importance of incorporating non-human perspectives to ensure safer, more inclusive futures for all species.

cs.HC

Enhancing Autonomous Vehicle-Pedestrian Interaction in Shared Spaces: The Impact of Intended Path-Projection

External Human-Machine Interfaces (eHMIs) are critical for seamless interactions between autonomous vehicles (AVs) and pedestrians in shared spaces. However, they often struggle to adapt to these environments, where pedestrian movement is fluid and right-of-way is ambiguous. To address these challenges, we propose PaveFlow, an eHMI that projects the AV's intended path onto the ground in real time, providing continuous spatial information rather than a binary stop/go signal. Through a VR study (N=18), we evaluated PaveFlow's effectiveness under two AV density conditions (single vs. multiple AVs) and a baseline condition without PaveFlow. The results showed that PaveFlow significantly improved pedestrian perception of safety, trust, and user experience while reducing cognitive workload. This performance remained consistent across both single and multiple AV conditions, despite persistent tensions in priority negotiation. These findings suggest that path projection enhances eHMI transparency by offering richer movement cues, which may better support AV-pedestrian interaction in shared spaces.

cs.HC

Peek into the `White-Box': A Field Study on Bystander Engagement with Urban Robot Uncertainty

Uncertainty inherently exists in the autonomous decision-making process of robots. Involving humans in resolving this uncertainty not only helps robots mitigate it but is also crucial for improving human-robot interactions. However, in public urban spaces filled with unpredictability, robots often face heightened uncertainty without direct human collaborators. This study investigates how robots can engage bystanders for assistance in public spaces when encountering uncertainty and examines how these interactions impact bystanders' perceptions and attitudes towards robots. We designed and tested a speculative `peephole' concept that engages bystanders in resolving urban robot uncertainty. Our design is guided by considerations of non-intrusiveness and eliciting initiative in an implicit manner, considering bystanders' unique role as non-obligated participants in relation to urban robots. Drawing from field study findings, we highlight the potential of involving bystanders to mitigate urban robots' technological imperfections to both address operational challenges and foster public acceptance of urban robots. Furthermore, we offer design implications to encourage bystanders' involvement in mitigating the imperfections.

cs.RO

Robots in the Wild: Contextually-Adaptive Human-Robot Interactions in Urban Public Environments

The increasing transition of human-robot interaction (HRI) context from controlled settings to dynamic, real-world public environments calls for enhanced adaptability in robotic systems. This can go beyond algorithmic navigation or traditional HRI strategies in structured settings, requiring the ability to navigate complex public urban systems containing multifaceted dynamics and various socio-technical needs. Therefore, our proposed workshop seeks to extend the boundaries of adaptive HRI research beyond predictable, semi-structured contexts and highlight opportunities for adaptable robot interactions in urban public environments. This half-day workshop aims to explore design opportunities and challenges in creating contextually-adaptive HRI within these spaces and establish a network of interested parties within the OzCHI research community. By fostering ongoing discussions, sharing of insights, and collaborations, we aim to catalyse future research that empowers robots to navigate the inherent uncertainties and complexities of real-world public interactions.

cs.RO

Encouraging Bystander Assistance for Urban Robots: Introducing Playful Robot Help-Seeking as a Strategy

Robots in urban environments will inevitably encounter situations beyond their capabilities (e.g., delivery robots unable to press traffic light buttons), necessitating bystander assistance. These spontaneous collaborations possess challenges distinct from traditional human-robot collaboration, requiring design investigation and tailored interaction strategies. This study investigates playful help-seeking as a strategy to encourage such bystander assistance. We compared our designed playful help-seeking concepts against two existing robot help-seeking strategies: verbal speech and emotional expression. To assess these strategies and their impact on bystanders' experience and attitudes towards urban robots, we conducted a virtual reality evaluation study with 24 participants. Playful help-seeking enhanced people's willingness to help robots, a tendency more pronounced in scenarios requiring greater physical effort. Verbal help-seeking was perceived less polite, raising stronger discomfort assessments. Emotional expression help-seeking elicited empathy while leading to lower cognitive trust. The triangulation of quantitative and qualitative results highlights considerations for robot help-seeking from bystanders.

cs.HC

From Agent Autonomy to Casual Collaboration: A Design Investigation on Help-Seeking Urban Robots

As intelligent agents transition from controlled to uncontrolled environments, they face challenges that sometimes exceed their operational capabilities. In many scenarios, they rely on assistance from bystanders to overcome those challenges. Using robots that get stuck in urban settings as an example, we investigate how agents can prompt bystanders into providing assistance. We conducted four focus group sessions with 17 participants that involved bodystorming, where participants assumed the role of robots and bystander pedestrians in role-playing activities. Generating insights from both assumed robot and bystander perspectives, we were able to identify potential non-verbal help-seeking strategies (i.e., addressing bystanders, cueing intentions, and displaying emotions) and factors shaping the assistive behaviours of bystanders. Drawing on these findings, we offer design considerations for help-seeking urban robots and other agents operating in uncontrolled environments to foster casual collaboration, encompass expressiveness, align with agent social categories, and curate appropriate incentives.

cs.HC