Search arXiv⌕ Search
Home/Archive/arXiv

arXiv · 2003.02836

Guided Generative Adversarial Neural Network for Representation Learning and High Fidelity Audio Generation using Fewer Labelled Audio Data

Kazi Nazmul Haque·Rajib Rana·John H. L. Hansen·Björn Schuller
View original source ↗Read source document ↗Export metadata ↓

Abstract

Recent improvements in Generative Adversarial Neural Networks (GANs) have shown their ability to generate higher quality samples as well as to learn good representations for transfer learning. Most of the representation learning methods based on GANs learn representations ignoring their post-use scenario, which can lead to increased generalisation ability. However, the model can become redundant if it is intended for a specific task. For example, assume we have a vast unlabelled audio dataset, and we want to learn a representation from this dataset so that it can be used to improve the emotion recognition performance of a small labelled audio dataset. During the representation learning training, if the model does not know the post emotion recognition task, it can completely ignore emotion-related characteristics in the learnt representation. This is a fundamental challenge for any unsupervised representation learning model. In this paper, we aim to address this challenge by proposing a novel GAN framework: Guided Generative Neural Network (GGAN), which guides a GAN to focus on learning desired representations and generating superior quality samples for audio data leveraging fewer labelled samples. Experimental results show that using a very small amount of labelled data as guidance, a GGAN learns significantly better representations.

Explore related subjects

eess.AScs.LGcs.SDstat.ML

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Kazi Nazmul Haque, Rajib Rana, John H. L. Hansen, Björn Schuller. 2020-06-01. Guided Generative Adversarial Neural Network for Representation Learning and High Fidelity Audio Generation using Fewer Labelled Audio Data. https://arxiv.org/abs/2003.02836

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages

Phoneme-based multilingual automatic speech recognition (ASR) can share acoustic evidence across languages more directly than language-specific subword modeling. When tonal and non-tonal languages are jointly trained, however, their supervision granularity does not match: tonal languages annotate tone-marked vowels, whereas non-tonal languages typically provide only base-vowel labels. A standard softmax either treats the two as unrelated classes, weakening cross-lingual sharing, or collapses tones, losing distinctions required by tonal languages. We propose Latent Softmax, a connectionist temporal classification (CTC)-compatible output layer that models tone-marked vowels as subclasses and base vowels as major classes, while consonants and the CTC blank remain singleton labels. When only a base-vowel major-class label is observed, the tone-marked vowel subclass is treated as latent and marginalized out. Multilingual experiments on AISHELL-1 Mandarin and LibriSpeech English show that Latent Softmax reduces speech-to-phoneme (S2P) phoneme error rates over a standard softmax multilingual baseline by 8.4% on AISHELL-1, 17.5% on LibriSpeech test-clean, and 12.6% on test-other. The improved speech-to-phoneme encoders also yield consistent word error rate gains for both large-language-model phoneme-to-grapheme conversion and projector-based interfaces. After code-switching adaptation, Latent Softmax further reduces projector-based mixed error rate by 2.6% on ASRU2019 and 9.5% on CS-Dialogue datasets.

eess.AS↗
arXiv2026

Learned Bow Control on a Measured Bowed-String Model: a Revised Minimum-Bow-Force Law, a Recurrent Controller, and the Domain of a Supervision Ceiling

A finite-difference bowed-string model with implicitly resolved Stribeck friction is presented, with a regime diagnostic, the Schelleng bow-force limits on four strings, and a comparison of learned bow controllers. Implicit resolution is necessary, and quantitatively so: a lagged contact force cannot capture the string on a discrete grid, so no stick phase forms at any bow force. With friction, impedance and quality factor taken from published measurement rather than fitted, all four strings return a stick fraction of 89.1% against an ideal 90%. Schelleng's maximum bow force is recovered on every string. The minimum is not: it follows $Z v_b β^{-1}$ rather than the predicted $Z^2 v_b β^{-2}$, reducing both squared dependences to first powers. Six controllers at matched capacity, over four strings and twenty seeds each, place a gated recurrent network ahead of a feedforward one, by most under a mid-stroke disturbance. The feedforward network completes more strokes only from a start the model's own playability map places outside the Helmholtz region. A minimal gated variant fails because gates computed from the input alone cannot clear a latched state. Training loss selects neither the capacity nor the context length, and no learned controller improves on the lookup rule that generated its labels. That bound has a domain. Regressing the controller's score on the rule's gives a slope of 0.32, more than ten standard errors below unity, so the controller overtakes the rule where the rule fails and is bounded by it where it holds. Under a rigid finger stop the plant is provably invariant, so transfer loss between pitches belongs to the controller alone and is traced to one feature. A regime classifier without a stick test labels small-amplitude periodic slipping as Helmholtz motion, and a harmonicity measure rates a string the bow never grips above Helmholtz motion.

eess.AS↗
arXiv2026

LLMs and Speech: Integration vs. Combination

In this work, we study different approaches to utilize large language models (LLMs) for automatic speech recognition (ASR). Specifically, we compare the tight integration of an acoustic model (AM) with the LLM ("speech LLM") to the traditional way of combining AM and LLM via shallow fusion and provide ablations on the effect of different label units and LLM sizes. For tight integration, we further examine the effect of attention interfaces, encoder downsampling, and length normalization. Furthermore, we investigate joint recognition with a CTC model to mitigate hallucinations of speech LLMs and present effective optimizations. We train and evaluate on LibriSpeech and Loquacious and additionally evaluate on the HuggingFace ASR leaderboard. Across model sizes, we find that shallow fusion consistently outperforms tight integration of AM and LLM on in-domain data, highlighting the importance of strong shallow-fusion baselines when evaluating speech LLMs for ASR. On the more heterogeneous HuggingFace ASR leaderboard, however, the integrated prefix LLM achieves lower average WER than shallow fusion, with gains concentrated on out-of-domain corpora.

eess.AS↗