RA-CLAP: Relation-Augmented Emotional Speaking Style Contrastive Language-Audio Pretraining For Speech Retrieval

Haoqin Sun; Jingguang Tian; Jiaming Zhou; Hui Wang; Jiabei He; Shiwan Zhao; Xiangyu Kong; Desheng Hu; Xinkang Xu; Xinhui Hu; Yong Qin

arXiv · 2505.19437

RA-CLAP: Relation-Augmented Emotional Speaking Style Contrastive Language-Audio Pretraining For Speech Retrieval

Haoqin Sun·Jingguang Tian·Jiaming Zhou·Hui Wang·Jiabei He·Shiwan Zhao·Xiangyu Kong·Desheng Hu·Xinkang Xu·Xinhui Hu·Yong Qin

View original source ↗Read source document ↗Export metadata ↓

Abstract

The Contrastive Language-Audio Pretraining (CLAP) model has demonstrated excellent performance in general audio description-related tasks, such as audio retrieval. However, in the emerging field of emotional speaking style description (ESSD), cross-modal contrastive pretraining remains largely unexplored. In this paper, we propose a novel speech retrieval task called emotional speaking style retrieval (ESSR), and ESS-CLAP, an emotional speaking style CLAP model tailored for learning relationship between speech and natural language descriptions. In addition, we further propose relation-augmented CLAP (RA-CLAP) to address the limitation of traditional methods that assume a strict binary relationship between caption and audio. The model leverages self-distillation to learn the potential local matching relationships between speech and descriptions, thereby enhancing generalization ability. The experimental results validate the effectiveness of RA-CLAP, providing valuable reference in ESSD.

Explore related subjects

cs.SD eess.AS

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Haoqin Sun, Jingguang Tian, Jiaming Zhou, Hui Wang, Jiabei He, Shiwan Zhao, Xiangyu Kong, Desheng Hu, Xinkang Xu, Xinhui Hu, Yong Qin. 2025-05-26. RA-CLAP: Relation-Augmented Emotional Speaking Style Contrastive Language-Audio Pretraining For Speech Retrieval. https://arxiv.org/abs/2505.19437

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Causal Tracing of Audio-Text Fusion in Large Audio Language Models

Despite the strong performance of large audio language models (LALMs) in various tasks, exactly how and where they integrate acoustic features with textual context remains unclear. We adapt causal tracing to investigate the internal information flow of LALMs during audio comprehension. By conducting layer-wise and token-wise analyses across DeSTA, Qwen, and Voxtral, we evaluate the causal effects of individual hidden states. Layer-wise analysis identifies different fusion strategies, from progressive integration in DeSTA to abrupt late-stage fusion in Qwen. Token-wise analysis shows that the final sequence token acts as an informational bottleneck where the network decisively retrieves relevant information from the audio. We also observe an attention-like query mechanism at intermediate token positions that triggers the model to pull task-relevant audio context. These findings provide a clear characterization of when and where multi-modal integration occurs within LALMs.

cs.SD↗

arXiv2026

Sona: Personalized Soundscape Mediation to Support People with Sound Sensitivity

People with sound sensitivity (PWSS) often manage distressing sounds with earplugs and noise-canceling headphones that broadly suppress their surroundings, limiting access to useful auditory cues. We present Sona, a mobile system for personalized, real-time soundscape mediation, informed by prior sound sensitivity research and an online survey of 68 PWSS. Sona selectively attenuates multiple overlapping user-chosen sounds at adjustable strength, suggests targets from ambient sound recognition, and lets users add custom targets from short recordings without retraining the model. In an in-situ evaluation with ten PWSS, participants reported that Sona made their soundscapes more manageable. The study also surfaced uneven attenuation across sound type and context, tensions between managing filters and attending to ongoing activities, and difficulty interpreting personalization outcomes. These findings highlight the need to design for the quality of the residual soundscape, balance user control with interaction demands, and support guided, interpretable personalization.

cs.SD↗

arXiv2026

FRAUDSkill: Structured Frozen-Weight Skill Optimization for Audio Anti-Fraud Detection

Large audio-language models have shown promise for anti-fraud detection by directly processing speech and reasoning over fraud-related evidence. Their deployment, however, requires predictions to follow a predefined label space and a structured decision protocol consisting of service-scenario identification, fraud detection, and conditional fraud-type classification. Existing fine-tuning and prompt-based approaches typically encode task knowledge, constraints, and decision rules into model parameters or manually maintained prompts, making them difficult to adapt as fraud patterns and labeling policies evolve. To this end, we propose FRAUDSkill, a structured frozen-weight adaptation framework that leaves the underlying audio-language model unchanged while optimizing an external layer of skill programs, route-specific policies, and decision rules. We further combine structured output control with validation-guided multi-path inference to ensure protocol-compliant predictions. On the TeleAntiFraud benchmark, FRAUDSkill achieves 73.50% Macro-F1, outperforming the shared frozen-model baseline by 31.96% while reducing invalid outputs to 1.94%. Extensive experiments demonstrate that external skill optimization provides an effective and adaptable solution for structured audio anti-fraud detection without modifying the underlying model. The source code is available at https://anonymous.4open.science/r/FRAUDSKILL-114514.

cs.SD↗