Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.29836

AT-Differ: Dataset for Audio-Text Difference Captioning

Akihiro Fujii·Yuki Okamoto·Yuki Saito·Hiroshi Saruwatari
View original source ↗Read source document ↗Export metadata ↓

Abstract

Evaluating audio-text semantic alignment is important in various audio-language tasks. Existing methods mainly quantify audio-text semantic alignment using a single score and cannot explicitly describe how audio and text differ. In this paper, we propose audio-text difference captioning, a task that generates textual descriptions of differences between audio and text. We also construct a new dataset AT-DIFFER containing audio-text pairs with systematically simulated differences and corresponding difference captions. Using AT-DIFFER, we fine-tune an existing multimodal large language model to describe differences between audio and text. The experimental results show that the original model has limited ability to describe such differences, whereas fine-tuning on AT-DIFFER substantially improves this ability.

Explore related subjects

cs.SD

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Akihiro Fujii, Yuki Okamoto, Yuki Saito, Hiroshi Saruwatari. 2026-10-21. AT-Differ: Dataset for Audio-Text Difference Captioning. https://arxiv.org/abs/2610.29836

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS

Text encoders in neural text-to-speech (TTS) see only linguistic content, so prosody must be inferred from paired data and synthesised intonation is often flat. Contrastive text-audio pretraining (CLAPSpeech) injects prosodic knowledge by contrasting acoustic realisations of the same phoneme, but it never trains the encoder to tell different phonemes apart, and its interaction with masked language modelling (MLM) is unexplored. We study a curriculum for a speaker-conditioned phoneme-and-subword encoder: MLM, contrastive alignment over batches of different phonemes, and CLAPSpeech-style same-phoneme refinement. On Russian conversational speech, with each variant conditioning an identical Grad-TTS model, MLM followed by mixed-phoneme alignment gives the lowest word error rate (17.6% versus 32.8% for same-phoneme training alone) and the best quality and intonation ratings. Adding same-phoneme refinement improves same-phoneme retrieval yet yields the worst synthesis: embedding-space retrieval does not predict usefulness for generation.

cs.SD↗
arXiv2026

Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Model (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors are more sensitive to acoustic signal-level properties at the prosodic and paralinguistic levels, while Audio-LLM judges show selective, prompt-dependent detection that favors word-level errors and does not generalize across all dimensions. Neither class reliably captures a breadth of linguistically structured, human-perceptible speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.

cs.SD↗
arXiv2026

Qwen-Audio-Jev: A Unified Interface for Audio Decisions

Large audio language models (LALMs) have made substantial progress in audio understanding. However, autoregressive text generation introduces computational overhead and latency in high-throughput applications that require frequent selection, judgment, and scoring. To address this limitation, we introduce Qwen-Audio-Jev, which augments Qwen-Audio-3.1-ASR-Flash-Next with a dedicated Jev decision head. The model unifies selection, binary judgment, and scoring through three decision primitives: CHOICE, NOUL, and SCORE. Given an audio input, a natural-language question, and an explicit answer specification, Qwen-Audio-Jev directly outputs a task-conditioned probability distribution or a numerical score through typed decision readouts. Evaluation across 18 benchmarks demonstrates the model's effectiveness: CHOICE achieves 78.40 percent accuracy on MMAU and 83.14 percent on MMSU; NOUL achieves 93.76 percent accuracy on the VoxCeleb1-O speaker verification benchmark; and SCORE achieves a normalized mean absolute error of 0.1235 on NISQA overall quality assessment. These results highlight the potential of Qwen-Audio-Jev for efficient audio decision-making in applications such as large-scale audio screening, speaker verification, and automated quality assessment.

cs.SD↗