End-to-end feature fusion for jointly optimized speech enhancement and automatic speech recognition

Сквозное слияние признаков для совместной оптимизации улучшения речи и автоматического распознавания речи
Sami Bourouis, Hela Elmannai, Nasir Saleem, Mohamed Medani, Fethi Fkih, Manal Abdullah Alohali
2025-07-01

GRU-based fusion networkautomatic speech recognitioncausal attentiondynamic feature fusionspeech enhancement
Speech enhancement (SE) and automatic speech recognition (ASR) in real-time processing involve improving the quality and intelligibility of speech signals on the fly, ensuring accurate transcription as the speech unfolds. SE eliminates unwanted background noise from target speech in environments with high background noise levels, which is crucial in real-time ASR. This study first proposes a speech enhancement network based on an attentional-codec model. Its primary objective is to suppress noise in the target speech with minimal distortion. However, excessive noise suppression in the enhanced speech can potentially diminish the effectiveness of downstream ASR systems by excluding crucial latent information. While joint SE and ASR techniques have shown promise for achieving robust end-to-end ASR, they traditionally rely on using the enhanced features as inputs to the ASR systems. To address this limitation, our study uses a dynamic fusion approach. This approach integrates both the enhanced features and the raw noisy features, aiming to eliminate noise signals from the enhanced target speech while simultaneously learning fine details from the noisy signals. This fusion approach seeks to mitigate speech distortions, enhancing the overall performance of the ASR system. The proposed model consists of an attentional codec equipped with a causal attention mechanism for SE, a GRU-based fusion network, and an ASR system. The SE network uses a modified Gated Recurrent Unit (GRU), where the traditional hyperbolic tangent (tanh) is replaced by an attention-based rectified linear unit (AReLU). The SE experiments consistently obtain better speech quality, intelligibility, and noise suppression in matched and unmatched conditions than the baselines. With the LibriSpeech database, the proposed SE obtains better STOI (19.81%) and PESQ (28.97%) in matched conditions and unmatched conditions (STOI: 17.27% and PESQ: 27.51%). The joint training framework for robust end-to-end ASR evaluates the character error rate (CER). The ASR results find that the joint training framework reduces the error rate from 32.99% (average noisy signals) to 13.52% (with the proposed SE and joint training for ASR).
1
A dynamic fusion approach combines enhanced and raw noisy features, preserving fine-grained information that may be lost through excessive noise suppression.
2
Compared with baselines, the proposed enhancement system consistently improves speech quality, intelligibility, and noise suppression in matched and unmatched conditions, including STOI gains of 19.81% and 17.27% and PESQ gains of 28.97% and 27.51%, respectively.
3
Replacing tanh with attention-based AReLU in the modified GRU improves the speech enhancement architecture’s ability to model relevant features.
4
The joint model integrates causal-attention speech enhancement, a GRU-based fusion network, and an end-to-end ASR system.
5
The study proposes an attentional-codec speech enhancement network designed to suppress noise while minimizing distortion in target speech.

joint speech enhancement and automatic speech recognition system processing noisy speech signals

dynamic end-to-end fusion of enhanced and raw noisy speech features to improve speech quality, intelligibility, noise suppression, and robust ASR performance

Publication Details
Publication Date
2025-07-01
Journal
Publisher
ISSN
Cited by
11
Access Type
Author Information
Authors
Sami Bourouis
Hela Elmannai
Nasir Saleem
Mohamed Medani
Fethi Fkih
Manal Abdullah Alohali
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%