Recurrent neural network based language model

Модель языка на основе рекуррентной нейронной сети
Tomáš Mikolov, Sanjeev Khudanpur, Martin Karafiát, Lukáš Burget, Jaň Černocký
2010-09-26
SCID:  54.1/5k3k6bzh Discuss with AI
NIST RT05 taskRNN LMWall Street Journal speech recognitionbackoff language modelconnectionist language modelsmixture of RNN LMsperplexity reductionrecurrent neural network based language modelword error rate reduction
Abstract (AI)
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Key Findings
1
A recurrent neural network language model (RNN LM) is presented for speech recognition applications.
2
Empirical evidence suggests connectionist (RNN) language models outperform standard n-gram techniques, with the primary drawback being high computational training complexity.
3
Mixtures of several RNN LMs can achieve around 50% reduction in perplexity compared to a state-of-the-art backoff language model.
4
On the NIST RT05 task, RNN LMs produce about a 5% reduction in word error rate, even when backoff models are trained on much more data.
5
On the Wall Street Journal task, RNN LMs trained on the same data yield about an 18% reduction in word error rate versus backoff models.
Research Object

Recurrent neural network based language model (RNN LM)

Research Subject

Language modeling performance and its impact on speech recognition (perplexity reduction and word error rate reduction) achieved by RNN-based LMs including mixtures compared to backoff n-gram models, and associated training computational complexity

Publication Details
Publication Date
2010-09-26
Journal
Publisher
ISSN
Access Type
Author Information
Authors
Tomáš Mikolov
Sanjeev Khudanpur
Martin Karafiát
Lukáš Burget
Jaň Černocký
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%