Evaluating LLM-based Agents for Multi-turn Conversations: A Survey

Оценка агентов на основе больших языковых моделей в многоходовых диалогах: обзор
Jindong Wang, 将尚 渡辺, Haoyi Xiong, Jian–Guang Lou, Shengyue Guan, Bin Zhu
2026-02-14

LLM-based agentsevaluation methodologiesmemory and context retentionmulti-turn conversationsself-judging methods
This survey examines evaluation methods for large language model (LLM)-based agents in multi-turn conversational settings. Using a PRISMA-inspired framework, we systematically reviewed nearly 250 scholarly sources, capturing the state-of-the-art from various venues of publication, and establishing a solid foundation for our analysis. Our study offers a structured approach by developing two interrelated taxonomy systems: one that defines what to evaluate and another that explains how to evaluate . The first taxonomy identifies key components of LLM-based agents for multi-turn conversations and their evaluation dimensions, including task completion, response quality, user experience, memory and context retention, as well as planning and tool integration. These components ensure that the performance of conversational agents is assessed in a holistic and meaningful manner. The second taxonomy system focuses on the evaluation methodologies. It categorizes approaches into annotation-based evaluations, automated metrics, hybrid strategies that combine human assessments with quantitative measures, and self-judging methods utilizing LLMs. This framework not only captures traditional metrics derived from language understanding, such as BLEU and ROUGE scores, but also incorporates advanced techniques that reflect the dynamic, interactive nature of multi-turn dialogues. Together, these frameworks summarize the current status quo, expose limitations in traditional practices, and provide a structured blueprint for improvement. Based on the summarization of existing studies, we identify several challenges and propose future directions, including the development of scalable, real-time evaluation pipelines, enhanced privacy-preserving mechanisms, and robust metrics that capture dynamic multi-turn interactions. Our contributions bridge historical insights with modern practices, paving the way for next-generation, reliably evaluated conversational AI systems and offering a comprehensive guide for researchers and practitioners.
1
A complementary taxonomy organizes evaluation methods into annotation-based, automated-metric, hybrid human-quantitative, and LLM self-judging approaches.
2
A taxonomy of evaluation targets covers task completion, response quality, user experience, memory and context retention, planning, and tool integration.
3
The framework shows that traditional language metrics such as BLEU and ROUGE are insufficient to fully capture dynamic, interactive multi-turn dialogue.
4
The survey identifies priorities including scalable real-time evaluation, privacy-preserving mechanisms, and robust metrics for dynamic multi-turn interactions.
5
The survey systematically reviews nearly 250 scholarly sources on evaluating LLM-based agents in multi-turn conversational settings.

LLM-based agents in multi-turn conversational settings

Evaluation of their task completion, response quality, user experience, memory and context retention, planning, and tool integration using conversational evaluation methodologies

Publication Details
Publication Date
2026-02-14
Journal
Publisher
ISSN
Cited by
27
Access Type
Author Information
Authors
Jindong Wang
将尚 渡辺
Haoyi Xiong
Jian–Guang Lou
Shengyue Guan
Bin Zhu
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%