Evaluating LLM-based Agents for Multi-turn Conversations: A Survey
Оценка агентов на основе больших языковых моделей в многоходовых диалогах: обзор
2026-02-14
SCID: 54.1/mhu8npce
Discuss with AI
LLM-based agentsevaluation methodologiesmemory and context retentionmulti-turn conversationsself-judging methods
Figures from the paper
Abstract (AI)
This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings. Using a PRISMA-inspired framework, we systematically reviewed nearly 250 scholarly sources, capturing the state-of-the-art from various venues of publication, and establishing a solid foundation for our analysis. Our study offers a structured approach by developing two interrelated taxonomy systems: one that defines what to evaluate and another that explains how to evaluate . The first taxonomy identifies key components of LLM-based agents for multi-turn conversations and their evaluation dimensions, including task completion, response quality, user experience, memory and context retention, as well as planning and tool integration. These components ensure that the performance of conversational agents is assessed in a holistic and meaningful manner. The second taxonomy system focuses on the evaluation methodologies. It categorizes approaches into annotation-based evaluations, automated metrics, hybrid strategies that combine human assessments with quantitative measures, and self-judging methods utilizing LLMs. This framework not only captures traditional metrics derived from language understanding, such as BLEU and ROUGE scores, but also incorporates advanced techniques that reflect the dynamic, interactive nature of multi-turn dialogues. Together, these frameworks summarize the current status quo, expose limitations in traditional practices, and provide a structured blueprint for improvement. Based on the summarization of existing studies, we identify several challenges and propose future directions, including the development of scalable, real-time evaluation pipelines, enhanced privacy-preserving mechanisms, and robust metrics that capture dynamic multi-turn interactions. Our contributions bridge historical insights with modern practices, paving the way for next-generation, reliably evaluated conversational AI systems and offering a comprehensive guide for researchers and practitioners.
Key Findings
1
A complementary taxonomy organizes evaluation methods into annotation-based, automated-metric, hybrid human-quantitative, and LLM self-judging approaches.
2
A taxonomy of evaluation targets covers task completion, response quality, user experience, memory and context retention, planning, and tool integration.
3
The framework shows that traditional language metrics such as BLEU and ROUGE are insufficient to fully capture dynamic, interactive multi-turn dialogue.
4
The survey identifies priorities including scalable real-time evaluation, privacy-preserving mechanisms, and robust metrics for dynamic multi-turn interactions.
5
The survey systematically reviews nearly 250 scholarly sources on evaluating LLM-based agents in multi-turn conversational settings.
Research Object
LLM-based agents in multi-turn conversational settings
Research Subject
Evaluation of their task completion, response quality, user experience, memory and context retention, planning, and tool integration using conversational evaluation methodologies
Publication Details
Publication Date
2026-02-14
Journal
Publisher
ISSN
Cited by
27
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai7
A Survey on Evaluation of Large Language Models2024
Training language models to follow instructions with human feedback2022
A survey on large language model based autonomous agents2024
Generative Agents: Interactive Simulacra of Human Behavior2023
GPT-4 Technical Report2023
Personalizing Dialogue Agents: I have a dog, do you have pets too?2018
CToolEval: A Chinese Benchmark for LLM-Powered Agent Evaluation in Real-World API Interactions2024