Recurrent neural network based language model
Модель языка на основе рекуррентной нейронной сети
2010-09-26
SCID: 54.1/5k3k6bzh
Discuss with AI
NIST RT05 taskRNN LMWall Street Journal speech recognitionbackoff language modelconnectionist language modelsmixture of RNN LMsperplexity reductionrecurrent neural network based language modelword error rate reduction
Figures from the paper
Abstract (AI)
A new recurrent neural network based language model (RNN LM) with applications to speech recognition is presented. Results indicate that it is possible to obtain around 50% reduction of perplexity by using mixture of several RNN LMs, compared to a state of the art backoff language model. Speech recognition experiments show around 18% reduction of word error rate on the Wall Street Journal task when comparing models trained on the same amount of data, and around 5% on the much harder NIST RT05 task, even when the backoff model is trained on much more data than the RNN LM. We provide ample empirical evidence to suggest that connectionist language models are superior to standard n-gram techniques, except their high computational (training) complexity. Index Terms: language modeling, recurrent neural networks, speech recognition
Key Findings
1
A recurrent neural network language model (RNN LM) is presented for speech recognition applications.
2
Empirical evidence suggests connectionist (RNN) language models outperform standard n-gram techniques, with the primary drawback being high computational training complexity.
3
Mixtures of several RNN LMs can achieve around 50% reduction in perplexity compared to a state-of-the-art backoff language model.
4
On the NIST RT05 task, RNN LMs produce about a 5% reduction in word error rate, even when backoff models are trained on much more data.
5
On the Wall Street Journal task, RNN LMs trained on the same data yield about an 18% reduction in word error rate versus backoff models.
Research Object
Recurrent neural network based language model (RNN LM)
Research Subject
Language modeling performance and its impact on speech recognition (perplexity reduction and word error rate reduction) achieved by RNN-based LMs including mixtures compared to backoff n-gram models, and associated training computational complexity
Publication Details
Publication Date
2010-09-26
Journal
Publisher
ISSN
Cited by
5535
Access Type
Author Information
Download PDF
Subscribe to digest
Cited by20
Sequence to Sequence Learning with Neural Networks2014
A Survey of Large Language Models2026
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context2019
Federated Learning in Mobile Edge Networks: A Comprehensive Survey2020
On the Opportunities and Risks of Foundation Models2021
BloombergGPT: A Large Language Model for Finance2023
🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models2025
Deep learning modelling techniques: current progress, applications, advantages, and challenges2023
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges2024
Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems2015
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation2018
Training Spiking Neural Networks Using Lessons From Deep Learning2023
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage2023
Deep learning in the construction industry: A review of present status and future innovations2020
Strategies for training large scale neural network language models2011
Spatio-Temporal Data Mining2018
Generative artificial intelligence in innovation management: A preview of future research developments2024
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech2021
Generative AI-Driven Semantic Communication Networks: Architecture, Technologies, and Applications2024
Large Language Models: A Comprehensive Survey of its Applications, Challenges, Limitations, and Future Prospects2023