Search arXiv⌕ Search

arXiv · 2611.09916

Semantic-Enhanced Sparse Temporal Modeling for Dynamic Engagement Estimation in Multimodal Interactions

Abstract

Dynamic engagement estimation aims to infer participants' evolving engagement states from continuous multimodal interaction streams. Existing approaches typically combine acoustic, pose, and visual representations with temporal models to capture observable interaction patterns. However, conversational semantics are difficult to align with frame-level engagement annotations in the same manner as continuous audio-visual signals, leaving dialogue context less effectively exploited in existing temporal modeling pipelines. Moreover, engagement annotations often evolve smoothly over time, making dense frame-level modeling computationally redundant and potentially sensitive to local annotation noise. To address these issues, we propose SEST, a Semantically-Enhanced Sparse Temporal framework that models engagement-related information from both semantic and temporal perspectives. SEST leverages Multimodal Large Language Models to extract multimodal semantic embeddings, providing dialogue-aware contextual information beyond conventional frame-level features. In parallel, SEST replaces dense-to-dense temporal modeling with a sparse-to-dense pathway, where continuous frame sequences are first encoded into compact temporal dynamics to reduce redundant computation and broaden temporal coverage, then fused with semantic context and recovered to the original label resolution for dense engagement prediction. Experiments on the MEE2026 benchmark demonstrate the effectiveness of SEST, which achieved first place in the challenge.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yangchen Yu, Taoliang Wang, Jia Li, Nor Mazlina Ghazali, Mohd Nazim Mohtar, Bingkun Bao, Richang Hong. 2026-08-29. Semantic-Enhanced Sparse Temporal Modeling for Dynamic Engagement Estimation in Multimodal Interactions. https://arxiv.org/abs/2611.09916

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

HexTiles and Semantic Icons for MAUP-Aware Multivariate Geospatial Visualizations

We introduce HexTiles, a hexagonal-tiling based visual encoding design for multivariate geospatial data. Visualizing several geospatial variables at once is difficult because map position is already reserved for geography, and because variables are often reported on different, irregular spatial units whose aggregation is subject to the Modifiable Areal Unit Problem (MAUP). HexTiles bins data of differing spatial discretizations into hierarchical hexagons and encodes up to three variables per tile in a base color, an inner ring, and semantic icons whose count conveys value, letting the visualization designer guide attention to specific variables and their interactions. To make the effects of MAUP visible, we compute a spatially-weighted variance for each variable within each tile and encode it as a confidence value in every channel, so that the variability hidden by aggregation is not lost. Because interactive maps are increasingly viewed in tilted, 3D perspective, HexTiles encodes quantitative values only in channels that perspective does not distort, namely color and icon count. In a user study comparing HexTiles to Square-glyphs, participants rated HexTiles as significantly less demanding and frustrating, and performed more consistently with it. Domain experts in ecology and hydrology further noted its value for communicating regional patterns, and that the confidence encoding signals where aggregated values should be interpreted with care. Our results suggest that, although 3D is commonly discouraged, it can work for multivariate geospatial visualization when restricted to perspective-robust channels.

cs.HC↗

AURA: A Reinforcement Learning Framework for AI-Driven Adaptive Conversational Surveys

Conventional online surveys provide limited personalization, often resulting in low engagement and superficial responses. Although AI survey chatbots improve convenience, most are still reactive: they rely on fixed dialogue trees or static prompt templates and therefore cannot adapt within a session to fit individual users, which leads to generic follow-ups and weak response quality. We address these limitations with AURA (Adaptive Understanding through Reinforcement Learning for Assessment), a reinforcement learning framework for AI-driven adaptive conversational surveys. AURA quantifies response quality using a four-dimensional LSDE metric (Length, Self-disclosure, Emotion, and Specificity) and selects follow-up question types via an epsilon-greedy policy that updates the expected quality gain within each session. Initialized with priors extracted from 96 prior campus-climate conversations (467 total chatbot-user exchanges), the system balances exploration and exploitation across 10-15 dialogue exchanges, dynamically adapting to individual participants in real time. In controlled evaluations, AURA achieved a +0.076 mean gain in response quality and a statistically significant improvement over non-adaptive baselines (p=0.044, d=0.66), driven by a 63% reduction in specification prompts and a 10x increase in validation behavior. These results demonstrate that reinforcement learning can give survey chatbots improved adaptivity, transforming static questionnaires into interactive, self-improving assessment systems.

cs.HC↗

Human-AI Collaboration in Large Language Model-Integrated Building Energy Management Systems: The Role of User Domain Knowledge and AI Literacy

This study aimed to comprehend how user domain knowledge and artificial intelligence (AI) literacy impact the effective use of human-AI interactive building energy management system (BEMS). While prior studies have investigated the potential of integrating large language models (LLMs) into BEMS or building energy modeling, very few studies have examined how user interact with such systems. We conducted a systematic role-playing experiment, where 85 human subjects interacted with an advanced generative pre-trained transformer (OpenAI GPT-4o). Participants were tasked with identifying the top five behavioral changes that could reduce home energy use with the GPT model that functioned as an LLM-integrated BEMS. Then, the collected prompt-response data and participant conclusions were analyzed using an analytical framework that hierarchically assessed and scored human-AI interactions and their home energy analysis approaches. Also, participants were classified into four groups based on their self-evaluated domain knowledge of building energy use and AI literacy, and Kruskal-Wallis H tests with post-hoc pairwise comparisons were conducted across 20 quantifiable metrics. Key takeaways include: most participants employed concise prompts (median: 16.2 words) and relied heavily on GPT's analytical capabilities; and notably, only 1 of 20 metrics, appliance identification rate, showed statistically significant group differences (p=0.037), driven by AI literacy rather than domain knowledge, suggesting an equalizing effect of LLMs across expertise levels. This study provides foundational insights into human-AI collaboration dynamics and promising development directions in the context of LLM-integrated BEMS and contributes to realizing human-centric LLM-integrated energy systems.

cs.HC↗