Towards Multilingual End-of-Turn Detection using Frozen Speech Encoders

Rini Sharon; Kadri Hacioglu; Andreas Stolcke

Home/Archive/arXiv

arXiv · 2610.39755

Towards Multilingual End-of-Turn Detection using Frozen Speech Encoders

Rini Sharon·Kadri Hacioglu·Andreas Stolcke

View original source ↗Read source document ↗Export metadata ↓

Abstract

A spoken dialogue system must judge when the user has yielded the floor. This end-of-turn (EoT) detector is normally trained and tuned afresh for each language, which is costly to scale to new languages. We instead build one on a frozen multilingual encoder, training only a small fixed-size head. Using five CALLHOME languages, we investigate three questions. First, how well a head trained on one language transfers zero-shot to the others. Second, how much target-language adaptation is required to recover full task performance. Third, whether a single fixed decision threshold holds across languages without per-language tuning. We find that the head transfers well, even between tonal and pitch-accent languages, and that a few hours of audio recover nearly all the loss, matching in-language training. A frozen multilingual encoder with one lightweight head is a practical, language-agnostic baseline for EoT detection.

Explore related subjects

eess.AS

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Rini Sharon, Kadri Hacioglu, Andreas Stolcke. 2026-10-29. Towards Multilingual End-of-Turn Detection using Frozen Speech Encoders. https://arxiv.org/abs/2610.39755

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Array-Agnostic High-Order Ambisonics Encoding via Diffusion Posterior Sampling

Ambisonics provides a device-independent representation of three-dimensional sound fields; however, extracting these coefficients from practical microphone array recordings remains strongly dependent on array geometry, acoustic transfer functions, and spatial sampling resolution. This pipeline is typically divided into two distinct tasks: encoding, which entails reconstructing the physically resolvable spatial components, and upscaling, which requires generating higher-order coefficients. Model-based approaches often fail on irregular arrays due to ill-conditioned transfer matrices, whereas conventional neural encoders remain strictly tied to the specific acquisition setup seen during training. Furthermore, existing upscaling methods usually assume ideal low-order inputs, ignoring device-specific array artifacts. We present ADEUPS, a unified framework that couples an array-independent Ambisonic diffusion prior with an explicit physical acquisition model during inference, enabling zero-shot encoding and upscaling across diverse array configurations. We train our prior using an order-masking strategy and perform inference via score blending, which we demonstrate to be crucial for solving the underlying inverse problem. Through extensive numerical evaluations across various array geometries, we show that ADEUPS consistently outperforms task- and array-specific baselines. Finally, we conduct subjective listening tests to confirm the perceptual effectiveness of our approach.

eess.AS↗

arXiv2026

Conformal Prediction-Aware DOA Tracking

We introduce an uncertainty-aware DOA-tracking framework combining deep posterior estimation, conformal-prediction calibration, and temporal smoothing. The approach enhances stability against reverberation and burst-like directional noise, yielding more reliable trajectories and greater robustness than conventional DOA-tracking methods.

eess.AS↗

arXiv2026

The Importance of Speech Genre for End-of-Turn Detector Training

End-of-turn (EoT) detection has been studied mostly through features, models, and modalities, while the speech genre of the training data has drawn far less attention. We isolate this factor. Using one reproducible annotation pipeline, we build a matched pair of corpora, multi-turn human-human conversational and single-turn human-machine, and train the same streaming ConvLSTM EoT detector on each. Across four test sets the conversational model reaches up to +0.45 higher F1, driven by fewer false alarms on the mid-turn pauses and hesitations that are rare in single-turn corpora. This conversational-training effect is not specific to frame-level decoding. Under a matched complete-or-not test it also appears at the utterance level, across two encoders, but only on conversational test data. Testing only on single-turn speech, as most prior work does, therefore hides the benefit of conversational training data. Against open-source baselines, our 1.8M-parameter model attains the highest F1 on every test set.

eess.AS↗