Multitask Transformer for Cross-Corpus Speech Emotion Recognition

Многозадачный трансформер для распознавания эмоций в речи в условиях межкорпусного переноса
Rajib Kumar Rana, Jagath Chandana Rajapakse, Carlos Busso, Chung-Soo Ahn
2025-01-07

contrastive learningcross-corpus speech emotion recognitiondecision-level fusionmultitask learningpre-trained transformer
Deep learning has significantly advanced the field of Speech Emotion Recognition (SER), yet its efficacy in cross-corpus scenarios remains a challenge. To overcome this limitation, recent studies demonstrate the success of multitask learning, which uses auxiliary tasks to reduce difference between source and target dataset (or transfer knowledge from source to target datasets). Despite the efforts, the overall accuracy for cross-corpus SER is still relatively low and needs attention. To improve performance, we propose a multitask framework with SER as the primary task and contrastive learning and information maximization as auxiliary tasks. We design the auxiliary tasks innovatively to use the target data without emotional labels to develop a better understanding of the target data. The core of our multitask framework is a pre-trained transformer. While transformers have gained attention in SER, their application to cross-corpus scenarios is still limited. Multimodal approaches for cross-corpus scenario is substantially limited as well. We use text as the second modality, developing separate multitask transformers for audio and text and conduct decision-level fusion during inference. We use publicly available and widely used speech corpora, including the IEMOCAP, MSP-IMPROV and EMO-DB databases. The results demonstrate the benefits of the proposed approach, achieving improved performance on the benchmark databases in cross-corpus settings.
1
Contrastive learning and information maximization use unlabeled target-domain data as auxiliary tasks to reduce cross-corpus differences and improve target understanding.
2
Experiments on IEMOCAP, MSP-IMPROV, and EMO-DB show improved performance on benchmark cross-corpus SER settings.
3
Separate multitask transformers process audio and text, with decision-level fusion combining the two modalities during inference.
4
The framework centers on a pre-trained transformer, addressing the limited use of transformers for cross-corpus SER.
5
The paper proposes a multitask framework for cross-corpus speech emotion recognition with SER as the primary task.

cross-corpus speech emotion recognition using audio and text from speech corpora

the performance and transferability of a multitask pre-trained transformer framework combining supervised emotion recognition with contrastive learning, information maximization, and decision-level multimodal fusion

Publication Details
Publication Date
2025-01-07
Journal
Publisher
ISSN
Cited by
23
Access Type
Author Information
Authors
Rajib Kumar Rana
Jagath Chandana Rajapakse
Carlos Busso
Chung-Soo Ahn
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%