End-to-end feature fusion for jointly optimized speech enhancement and automatic speech recognition
Сквозное слияние признаков для совместной оптимизации улучшения речи и автоматического распознавания речи
2025-07-01
SCID: 54.1/8xymee8k
Discuss with AI
GRU-based fusion networkautomatic speech recognitioncausal attentiondynamic feature fusionspeech enhancement
Figures from the paper
Abstract (AI)
Speech enhancement (SE) and automatic speech recognition (ASR) in real-time processing involve improving the quality and intelligibility of speech signals on the fly, ensuring accurate transcription as the speech unfolds. SE eliminates unwanted background noise from target speech in environments with high background noise levels, which is crucial in real-time ASR. This study first proposes a speech enhancement network based on an attentional-codec model. Its primary objective is to suppress noise in the target speech with minimal distortion. However, excessive noise suppression in the enhanced speech can potentially diminish the effectiveness of downstream ASR systems by excluding crucial latent information. While joint SE and ASR techniques have shown promise for achieving robust end-to-end ASR, they traditionally rely on using the enhanced features as inputs to the ASR systems. To address this limitation, our study uses a dynamic fusion approach. This approach integrates both the enhanced features and the raw noisy features, aiming to eliminate noise signals from the enhanced target speech while simultaneously learning fine details from the noisy signals. This fusion approach seeks to mitigate speech distortions, enhancing the overall performance of the ASR system. The proposed model consists of an attentional codec equipped with a causal attention mechanism for SE, a GRU-based fusion network, and an ASR system. The SE network uses a modified Gated Recurrent Unit (GRU), where the traditional hyperbolic tangent (tanh) is replaced by an attention-based rectified linear unit (AReLU). The SE experiments consistently obtain better speech quality, intelligibility, and noise suppression in matched and unmatched conditions than the baselines. With the LibriSpeech database, the proposed SE obtains better STOI (19.81%) and PESQ (28.97%) in matched conditions and unmatched conditions (STOI: 17.27% and PESQ: 27.51%). The joint training framework for robust end-to-end ASR evaluates the character error rate (CER). The ASR results find that the joint training framework reduces the error rate from 32.99% (average noisy signals) to 13.52% (with the proposed SE and joint training for ASR).
Key Findings
1
A dynamic fusion approach combines enhanced and raw noisy features, preserving fine-grained information that may be lost through excessive noise suppression.
2
Compared with baselines, the proposed enhancement system consistently improves speech quality, intelligibility, and noise suppression in matched and unmatched conditions, including STOI gains of 19.81% and 17.27% and PESQ gains of 28.97% and 27.51%, respectively.
3
Replacing tanh with attention-based AReLU in the modified GRU improves the speech enhancement architecture’s ability to model relevant features.
4
The joint model integrates causal-attention speech enhancement, a GRU-based fusion network, and an end-to-end ASR system.
5
The study proposes an attentional-codec speech enhancement network designed to suppress noise while minimizing distortion in target speech.
Research Object
joint speech enhancement and automatic speech recognition system processing noisy speech signals
Research Subject
dynamic end-to-end fusion of enhanced and raw noisy speech features to improve speech quality, intelligibility, noise suppression, and robust ASR performance
Publication Details
Publication Date
2025-07-01
Journal
Publisher
ISSN
Cited by
11
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest