arXiv · 2609.38440
Monotonicity-Guided Semantic Alignment for Zero-shot Multispeaker Image-to-Speech Synthesis
Abstract
Direct image-to-speech (Img2Sp) poses an alignment challenge in mapping visual content to ordered speech sequences, as images permit multiple spoken descriptions and lack monotonic correspondence with speech sequences. We propose Monotonicity-Guided Semantic Alignment (MGSA), to the best of our knowledge, the first framework for zero-shot multispeaker Img2Sp synthesis. We use semantic speech units to provide shared content targets across speakers with reference speech for speaker conditioning. A query aligner maps semantic memory learned from visual content to speech unit positions via a soft monotonic prior, which yields position-specific conditioning states. A blockwise masked diffusion generator is employed for the speech unit generation conditioning on these states. Experiments on Flickr8k-Audio show competitive captioning performance against single-speaker baselines, while evaluation with LibriTTS-R references supports zero-shot synthesis for unseen speakers. Ablations validate the effectiveness of aligner and block diffusion. Audio samples are available at https://alizeded.github.io/mgsa-demo.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Lijun Wang, Yixian Lu, Shogo Okada. 2026-09-29. Monotonicity-Guided Semantic Alignment for Zero-shot Multispeaker Image-to-Speech Synthesis. https://arxiv.org/abs/2609.38440
Cite the original work for its findings. Save a collection to share your selection of sources.