Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.31470

Measuring the Verifier's Floor: How Much Apparent Rare-Word Text-to-Speech Error Is ASR Error?

Kirill Borodin·Vasilii Kudryavtsev·Nikita Vasiliev·Grach Mkrtchian
View original source ↗Read source document ↗Export metadata ↓

Abstract

Text-to-speech (TTS) is usually evaluated by transcribing its output with an automatic speech recognition (ASR) system, so the word error rate also counts recognizer errors. We measure these errors on human recordings of the test sentences. On LibriSpeech and FLEURS, the evaluation recognizer misses 33-67% of very rare words even in human speech, and all three recognizers miss 22-42%. Checking every word with two more recognizer families lowers the apparent TTS error on words that at least one recognizer hears two- to three-fold (very rare LibriSpeech words: from 10-13% to 3-5%), yet very rare words remain the hardest. Ignoring words the recognizer misses in human speech is biased: the other recognizers find TTS errors far more often on these words (4-6% vs. 0.2-0.3%) and overturn about half of the errors this mask keeps. As half of the remaining adjudicated errors may be spelling variants, a listening test is needed.

Explore related subjects

cs.SDcs.CL

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Kirill Borodin, Vasilii Kudryavtsev, Nikita Vasiliev, Grach Mkrtchian. 2026-10-22. Measuring the Verifier's Floor: How Much Apparent Rare-Word Text-to-Speech Error Is ASR Error?. https://arxiv.org/abs/2610.31470

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS

Text encoders in neural text-to-speech (TTS) see only linguistic content, so prosody must be inferred from paired data and synthesised intonation is often flat. Contrastive text-audio pretraining (CLAPSpeech) injects prosodic knowledge by contrasting acoustic realisations of the same phoneme, but it never trains the encoder to tell different phonemes apart, and its interaction with masked language modelling (MLM) is unexplored. We study a curriculum for a speaker-conditioned phoneme-and-subword encoder: MLM, contrastive alignment over batches of different phonemes, and CLAPSpeech-style same-phoneme refinement. On Russian conversational speech, with each variant conditioning an identical Grad-TTS model, MLM followed by mixed-phoneme alignment gives the lowest word error rate (17.6% versus 32.8% for same-phoneme training alone) and the best quality and intonation ratings. Adding same-phoneme refinement improves same-phoneme retrieval yet yields the worst synthesis: embedding-space retrieval does not predict usefulness for generation.

cs.SD↗
arXiv2026

Beyond Naturalness: Probing Automated Text-To-Speech Evaluators on Linguistically Grounded Dimensions

Automated Text-to-Speech (TTS) evaluation methods (Mean Opinion Score (MOS) predictors and Audio Large Language Model (Audio-LLM) judges) are expected to reflect human perception, yet it is unclear how well they capture the distinct aspects of speech that listeners actually perceive. We deconstruct "naturalness" into a linguistically grounded annotation schema spanning 10 distinct perceptual dimensions, and use it to construct the first dimension-level meta-evaluation benchmark for TTS, comprising 860 utterances annotated by trained linguist raters. Results from benchmarking four MOS predictors and four Audio-LLM judges reveal that MOS predictors are more sensitive to acoustic signal-level properties at the prosodic and paralinguistic levels, while Audio-LLM judges show selective, prompt-dependent detection that favors word-level errors and does not generalize across all dimensions. Neither class reliably captures a breadth of linguistically structured, human-perceptible speech errors. Our dataset, annotation schema, and evaluation code are publicly released to support more targeted and interpretable TTS evaluation.

cs.SD↗
arXiv2026

Qwen-Audio-Jev: A Unified Interface for Audio Decisions

Large audio language models (LALMs) have made substantial progress in audio understanding. However, autoregressive text generation introduces computational overhead and latency in high-throughput applications that require frequent selection, judgment, and scoring. To address this limitation, we introduce Qwen-Audio-Jev, which augments Qwen-Audio-3.1-ASR-Flash-Next with a dedicated Jev decision head. The model unifies selection, binary judgment, and scoring through three decision primitives: CHOICE, NOUL, and SCORE. Given an audio input, a natural-language question, and an explicit answer specification, Qwen-Audio-Jev directly outputs a task-conditioned probability distribution or a numerical score through typed decision readouts. Evaluation across 18 benchmarks demonstrates the model's effectiveness: CHOICE achieves 78.40 percent accuracy on MMAU and 83.14 percent on MMSU; NOUL achieves 93.76 percent accuracy on the VoxCeleb1-O speaker verification benchmark; and SCORE achieves a normalized mean absolute error of 0.1235 on NISQA overall quality assessment. These results highlight the potential of Qwen-Audio-Jev for efficient audio decision-making in applications such as large-scale audio screening, speaker verification, and automated quality assessment.

cs.SD↗