LLaMA: Open and Efficient Foundation Language Models

Naman Goyal; Hugo Touvron; Marie-Anne Lachaux; Thibaut Lavril; Xavier Martinet; Aurelien Rodriguez; Guillaume Lample; Gautier Izacard; Édouard Grave; Armand Joulin; Timothée Lacroix; Baptiste Rozière; Eric Hambro; Faisal Azhar

doi:10.48550/arxiv.2302.13971

LLaMA: Open and Efficient Foundation Language Models

LLaMA: открытые и эффективные базовые языковые модели

Naman Goyal, Hugo Touvron, Marie-Anne Lachaux, Thibaut Lavril, Xavier Martinet, Aurelien Rodriguez, Guillaume Lample, Gautier Izacard, Édouard Grave, Armand Joulin, Timothée Lacroix, Baptiste Rozière, Eric Hambro, Faisal Azhar

2023-02-27

SCID: 54.1/vtr2pbdz Discuss with AI

GPT-3LLaMAfoundation language modelslanguage model benchmarkspublicly available datasets

Abstract (AI)

We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla-70B and PaLM-540B. We release all our models to the research community.

Key Findings

LLaMA introduces foundation language models spanning 7B to 65B parameters.

LLaMA-13B outperforms the much larger GPT-3 175B model on most benchmarks.

LLaMA-65B is competitive with Chinchilla-70B and PaLM-540B, despite having fewer parameters than PaLM-540B.

The authors release all LLaMA models to the research community.

The models are trained on trillions of tokens using only publicly available datasets, without proprietary or inaccessible data.

Research Object

LLaMA collection of foundation language models (7B–65B parameter models)

Research Subject

Their training efficiency and benchmark performance when trained exclusively on publicly available datasets

Publication Details

DOI

10.48550/arxiv.2302.13971

Publication Date

2023-02-27

Journal

Publisher

ISSN

Cited by

3973

Open access PDF

https://arxiv.org/pdf/2302.13971

Access Type

Author Information

Authors

Naman Goyal

Hugo Touvron

Marie-Anne Lachaux

Thibaut Lavril

Xavier Martinet

Aurelien Rodriguez

Guillaume Lample

Gautier Izacard

Édouard Grave

Armand Joulin

Timothée Lacroix

Baptiste Rozière

Eric Hambro

Faisal Azhar

Download PDF

Make a presentation

Cited by20

A Survey on Evaluation of Large Language Models2024 Lost in the Middle: How Language Models Use Long Contexts2024 The future landscape of large language models in medicine2023 Autonomous chemical research with large language models2023 A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges2024 From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy2023 Augmenting large language models with chemistry tools2024 Structured information extraction from scientific text with large language models2024 Large Language Models for Software Engineering: Survey and Open Problems2023 War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education2023 In-Context Retrieval-Augmented Language Models2023 GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints2023 Empowering biomedical discovery with AI agents2024 ChatMOF: an artificial intelligence system for predicting and generating metal-organic frameworks using large language models2024 Vision-language models for medical report generation and visual question answering: a review2024 Review of large vision models and visual prompt engineering2023 AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges2025 Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving2024 Automatic Generation of Multimedia Teaching Materials Based on Generative AI: Taking Tang Poetry as an Example2024 RenAIssance: A Survey Into AI Text-to-Image Generation in the Era of Large Model2024