JODAL: Joint Domain Adversarial Learning for TTS-Augmented Automatic Speech Recognition

June-Woo Kim; Ho-Young Jung;

HIGHLIGHT

Although we expect that training with both real and synthetic datasets together can mitigate the data scarcity issue while maintaining ASR performance, we observe that the distributional discrepancy between real and synthetic data introduces a domain inconsistency problem, which negatively impacts ASR performance. To mitigate this, we propose a Joint Domain Adversarial Learning (JODAL) method using real and TTS-generated samples that mitigates domain mismatches through domain adversarial training.

RELATED

Related Papers     Patents    Grants    Venues    Experts    Cite    Share    Save

ABSTRACT

Despite the rapid advancements in deep learning techniques, the scarcity of task-specific labeled data remains a significant challenge in automatic speech recognition (ASR). While data augmentation approaches and large-scale pretrained speech foundation models have shown promise in mitigating these limitations, they are often insufficient for improving downstream ASR performance in low-resource practical scenarios. We address these challenges by leveraging off-the-shelf text-to-speech (TTS) systems to generate synthetic speech samples using task-specific text and speech prompts in a resynthesis manner. Although we expect that training with both real and synthetic datasets together can mitigate the data scarcity issue while maintaining ASR performance, we observe that the distributional discrepancy between real and synthetic data introduces a domain inconsistency problem, which negatively impacts ASR performance. To mitigate this, we propose a Joint Domain Adversarial Learning (JODAL) method using real and TTS-generated samples that mitigates domain mismatches through domain adversarial training. Experimental results demonstrate that the proposed JODAL consistently improves ASR robustness across various architectures, including Whisper-tiny and three self-supervised models. Notably, for the WavLM architecture, the enhancement caused by the proposed JODAL successfully surpassed the performance of models trained on 100% real data, highlighting its efficacy as a reasonable substitute for expensive real-world data collection.

 Paraphrase     Questionfy     Wikify     Simplify     Summarize