Multitask Transformer for Cross-Corpus Speech Emotion Recognition
Многозадачный трансформер для распознавания эмоций в речи в условиях межкорпусного переноса
2025-01-07
SCID: 54.1/qjkfwqas
Discuss with AI
contrastive learningcross-corpus speech emotion recognitiondecision-level fusionmultitask learningpre-trained transformer
Figures from the paper
Abstract (AI)
Deep learning has significantly advanced the field of Speech Emotion Recognition (SER), yet its efficacy in cross-corpus scenarios remains a challenge. To overcome this limitation, recent studies demonstrate the success of multitask learning, which uses auxiliary tasks to reduce difference between source and target dataset (or transfer knowledge from source to target datasets). Despite the efforts, the overall accuracy for cross-corpus SER is still relatively low and needs attention. To improve performance, we propose a multitask framework with SER as the primary task and contrastive learning and information maximization as auxiliary tasks. We design the auxiliary tasks innovatively to use the target data without emotional labels to develop a better understanding of the target data. The core of our multitask framework is a pre-trained transformer. While transformers have gained attention in SER, their application to cross-corpus scenarios is still limited. Multimodal approaches for cross-corpus scenario is substantially limited as well. We use text as the second modality, developing separate multitask transformers for audio and text and conduct decision-level fusion during inference. We use publicly available and widely used speech corpora, including the IEMOCAP, MSP-IMPROV and EMO-DB databases. The results demonstrate the benefits of the proposed approach, achieving improved performance on the benchmark databases in cross-corpus settings.
Key Findings
1
Contrastive learning and information maximization use unlabeled target-domain data as auxiliary tasks to reduce cross-corpus differences and improve target understanding.
2
Experiments on IEMOCAP, MSP-IMPROV, and EMO-DB show improved performance on benchmark cross-corpus SER settings.
3
Separate multitask transformers process audio and text, with decision-level fusion combining the two modalities during inference.
4
The framework centers on a pre-trained transformer, addressing the limited use of transformers for cross-corpus SER.
5
The paper proposes a multitask framework for cross-corpus speech emotion recognition with SER as the primary task.
Research Object
cross-corpus speech emotion recognition using audio and text from speech corpora
Research Subject
the performance and transferability of a multitask pre-trained transformer framework combining supervised emotion recognition with contrastive learning, information maximization, and decision-level multimodal fusion
Publication Details
Publication Date
2025-01-07
Journal
Publisher
ISSN
Cited by
23
Access Type
Author Information
Download PDF
Subscribe to digest