A Survey on Evaluation of Large Language Models
Обзор методов оценки больших языковых моделей
2024-01-23
SCID: 54.1/ka2f7qhe
Discuss with AI
AI ethicsLLM evaluationevaluation benchmarkslarge language modelsreasoning evaluation
Figures from the paper
Abstract (AI)
Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate , where to evaluate , and how to evaluate . Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, education, natural and social sciences, agent applications, and other areas. Secondly, we answer the ‘where’ and ‘how’ questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing the performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey
Key Findings
1
It reviews evaluation tasks spanning general NLP, reasoning, medicine, ethics, education, natural and social sciences, and agent applications.
2
It summarizes both successful and failed LLM applications, highlighting the importance of understanding capabilities alongside limitations and risks.
3
The paper examines evaluation methods and benchmarks as central components for assessing LLM performance across diverse settings.
4
The survey identifies future challenges and argues that LLM evaluation should become an essential discipline supporting the development of more proficient models.
5
The survey organizes LLM evaluation around three dimensions: what to evaluate, where to evaluate, and how to evaluate.
Research Object
Large language models (LLMs)
Research Subject
evaluation of LLMs across tasks, domains, methods, benchmarks, and societal risks
Publication Details
Publication Date
2024-01-23
Journal
Publisher
ISSN
Cited by
6168
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai20
Exploiting Generative AI to Scale up Intelligent Tutoring Systems2023
ImageNet: A large-scale hierarchical image database2009
LLaMA: Open and Efficient Foundation Language Models2023
A Survey of Large Language Models2026
Large language models encode clinical knowledge2023
Training language models to follow instructions with human feedback2022
Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models2023
GPT-3: Its Nature, Scope, Limits, and Consequences2020
On the Opportunities and Risks of Foundation Models2021
Unifying Large Language Models and Knowledge Graphs: A Roadmap2024
PaLM: Scaling Language Modeling with Pathways2022
GPT-4 Technical Report2023
How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment2023
🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models2025
Sparks of Artificial General Intelligence: Early experiments with GPT-42023
Evaluating Large Language Models Trained on Code2021
Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios2023
Recommender Systems in the Era of Large Language Models (LLMs)2024
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models2022
Sentiment Analysis in the Era of Large Language Models: A Reality Check2024
Cited by20
A survey on large language model based autonomous agents2024
🧜Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models2025
Unleashing the potential of prompt engineering for large language models2025
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly2024
Large Language Models for Software Engineering: A Systematic Literature Review2024
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges2024
Benchmarking Large Language Models in Retrieval-Augmented Generation2024
Sycophantic AI decreases prosocial intentions and promotes dependence2026
Large language models in medical and healthcare fields: applications, advances, and challenges2024
A Survey on Large Language Models for Code Generation2025
Governance of Generative AI2025
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI2024
Automated data processing and feature engineering for deep learning and big data applications: A survey2024
A Survey on Trustworthy LLM Agents: Threats and Countermeasures2025
Toward Edge General Intelligence With Multiple-Large Language Model (Multi-LLM): Architecture, Trust, and Orchestration2025
Building LLM-based AI Agents in Social Virtual Reality2024
Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Review2024
Generative AI and LLMs for Critical Infrastructure Protection: Evaluation Benchmarks, Agentic AI, Challenges, and Opportunities2025
Large Language Models: A Structured Taxonomy and Review of Challenges, Limitations, Solutions, and Future Directions2025
A Research Landscape of Agentic AI and Large Language Models: Applications, Challenges and Future Directions2025