Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.25259

VoxDub: A Bilingual Lip-Synchronized Audio-Visual Dataset for Automatic Video Dubbing

Kaidi Wang·Wenhao Guan·Peijie Chen·Weijie Wu·Xuetao Lin·Lin Li·Qingyang Hong
View original source ↗Read source document ↗Export metadata ↓

Abstract

Automatic video dubbing (AVD) requires temporally aligned speech, transcripts, and video, yet existing corpora vary in audio quality, transcription reliability, and speaking-face visibility. We introduce VoxDub, a bilingual lip-synchronized audio-visual dataset containing 766K clips and approximately 1.58K hours of in-the-wild Chinese and English recordings, covering varied speakers, scenes, speaking styles, and linguistic content. Its curation pipeline combines speech-centered segmentation with dual-ASR transcription checking, speech quality filtering, face visibility checks, and lip-speech synchronization verification. Text-to-speech (TTS) experiments show that training on VoxDub yields lower word error rates and higher speaker similarity than training on the compared audio-visual corpora, LRS3 and CN-CVS. AVD experiments show broadly comparable speech quality and subjective synchronization. These results support the utility of VoxDub for TTS and AVD training. More details are available on the project website (https://voxdub-dataset.github.io/VoxDub/).

Explore related subjects

cs.SDeess.AS

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Kaidi Wang, Wenhao Guan, Peijie Chen, Weijie Wu, Xuetao Lin, Lin Li, Qingyang Hong. 2026-10-18. VoxDub: A Bilingual Lip-Synchronized Audio-Visual Dataset for Automatic Video Dubbing. https://arxiv.org/abs/2610.25259

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

TTS-Q: Speaker-Aware Post-Training Quantization of Text-to-Speech Diffusion Transformers

Zero-shot text-to-speech (TTS) can clone a voice from seconds of audio. Yet the quality comes from billion-parameter diffusion transformers whose size and multi-step sampling put them out of reach of the on-device, assistive, and real-time applications that need them most. Post-training quantization (PTQ) shrinks such models without retraining, but it has been built for language and vision, not speech. At 4 bits, a strong quantizer leaves intelligibility close to full precision but consistently erodes speaker timbre, so error rates stay low while the voice drifts from the target speaker. We ask why quantization hurts speech, where the damage lives, and whether it can be repaired without retraining. To find out, we run the first 4-bit weight-and-activation (W4A4) PTQ study of a TTS diffusion transformer at 1B and 3.5B. The timbre cost comes from both the weight and the activation quantization, but only the activation part has the timestep structure a precision schedule can exploit. We therefore introduce TTS-Q, which selects hard-to-quantize items from task-matched calibration candidates and applies a budget-balanced mixed-precision schedule that assigns higher precision to late-step feed-forward activations. TTS-Q recovers timbre best at both scales, at no cost to intelligibility. Separately, the deployed calibrated-W4A4 reference reduces peak memory on GPU, providing a practical reference for resource-constrained TTS deployment.

cs.SD↗
arXiv2026

MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning

Real-world audio typically contains concurrent sound events, yet standard self-supervised methods treat each clip as an isolated training unit without explicitly modeling mixture--constituent relationships. We propose Mixture-to-Constituent Latent Alignment (MiCLA), a two-stage self-supervised pre-training framework that uses synthetic log-mel mixing to align mixture representations with their constituent clips. Stage 1 structures single-clip representations via masked self-distillation and contrastive regularization. Stage 2 introduces bidirectional alignment: for a synthetic mixture, MiCLA suppresses the component aligned with one constituent and aligns the residual with the other, using projective residualization on patch tokens and sample-adaptive soft residualization on the global class token. In downstream evaluations, MiCLA achieves 43.2% mean average precision (mAP) on AS-20K and 50.3% mAP on AS-2M under fine-tuning, and achieves the highest mAP among evaluated models on both AudioSet settings under frozen Convex Gated Probing (CGP). Furthermore, a Joint Semantic Retrieval diagnostic shows that MiCLA improves simultaneous retrieval of both constituent classes over target-averaging baselines. These results indicate that directional mixture-to-constituent alignment provides an effective pre-training signal for polyphonic audio representation learning.

cs.SD↗
arXiv2026

PhonoBridge: Error-Aware Phoneme-Mediated EMG-to-Text Reconstruction

Electromyography (EMG) provides a non-invasive means of decoding silently articulated speech into text, but accurate decoding remains challenging due to limited availability of paired EMG-text data. We introduce PhonoBridge, a two-stage framework comprising a channel-aware EMG-to-phoneme frontend and a large language model (LLM) backend for text reconstruction. The frontend is trained on paired EMG-text data that consists of channel-wise feature extraction, cross-channel fusion, and electrode embeddings. With phonemes as the intermediate representation, we expand the backend's training data by converting LibriSpeech transcripts into phoneme-text pairs without additional EMG recordings. To adapt the LLM to frontend noise, we introduce Error-Profile-Guided Augmentation (EPGA), which perturbs transcript-derived phoneme sequences according to error patterns estimated from frontend predictions, generating additional training pairs for the LLM. On the Open Vocabulary silent-speech dataset, PhonoBridge achieves 24.77% WER, outperforming the previously reported best of 28.27% on the same test split.

cs.SD↗