From benchmarks to deployment: a comprehensive review of agentic AI evaluation
От бенчмарков к внедрению: всесторонний обзор оценки агентного искусственного интеллекта
2026-04-24
SCID: 54.1/jqusecpg
Discuss with AI
agent benchmarksagentic AI evaluationdeployment viabilitysafety-aware scoringtrajectory-level evaluation
Figures from the paper
Abstract (AI)
This review systematically examines evaluation methodologies for agentic AI systems, agentic AI systems capable of multi-step planning, tool usage, and environmental interaction across diverse domains. Current evaluation practices exhibit a critical disconnect between benchmark performance and deployment viability, where agents achieving high scores on standardized benchmarks frequently fail in real world applications due to fundamental inadequacies in assessment methodologies that prioritize task completion over deployment critical dimensions such as cost efficiency, safety compliance, maintainability, and workflow integration. We critically analyze 15 major agent benchmarks including AgentBench, WebArena, SWE-bench, PaperBench, MLGym, BrowserGym, HumanEval, MBPP, GAIA, ToolBench, Terminal-Bench, Mind2Web, ALFWorld, BabyAI, and HotPotQA, examining their methodologies, metrics, datasets, and application domains with software development serving as our primary case study. Our analysis reveals that evaluation methodology not model capability constitutes the primary bottleneck limiting reliable agent deployment. We demonstrate how test passing metrics systematically ignore code quality, security vulnerabilities, and integration complexity while binary success metrics obscure planning coherence, resource efficiency, and safety violations (0/15 benchmarks integrate security or safety into scoring. This review provides a cross-domain taxonomy exposing evaluation inadequacies, trajectory level evaluation frameworks addressing cost reproducibility validity trade-offs, systematic identification of metric insufficiencies including absence of safety-aware and cost aware scoring, and synthesis of emerging evaluation paradigms with adoption barriers. Quantitatively, 0/15 benchmarks integrate safety or security into scoring, 0/15 include cost-efficiency metrics in their primary evaluation protocol, and 13/15 rely exclusively on binary success measures confirming that evaluation methodology, not model capability, is the primary bottleneck to reliable deployment. Progress toward trustworthy agentic AI fundamentally depends on evolving evaluation infrastructure beyond binary metrics toward comprehensive, multidimensional assessment frameworks that capture these deployment-essential dimensions.
Key Findings
1
Across 15 major benchmarks, 0/15 integrate safety or security into scoring, 0/15 include cost-efficiency metrics, and 13/15 rely exclusively on binary success measures.
2
Binary task-success and test-passing metrics systematically overlook planning coherence, resource efficiency, code quality, security vulnerabilities, maintainability, and integration complexity.
3
Current agentic AI benchmarks show a critical disconnect between benchmark performance and deployment viability because they prioritize task completion over operational requirements.
4
The review identifies evaluation methodology—not model capability—as the primary bottleneck limiting reliable deployment of agentic AI systems.
5
The review proposes a cross-domain taxonomy and trajectory-level evaluation frameworks addressing cost, reproducibility, validity, safety-aware scoring, and cost-aware scoring, while identifying adoption barriers.
Research Object
agentic AI systems capable of multi-step planning, tool usage, and environmental interaction
Research Subject
evaluation methodologies and metrics for deployment viability, including task performance, cost efficiency, safety compliance, maintainability, and workflow integration
Publication Details
Publication Date
2026-04-24
Journal
Publisher
ISSN
Cited by
16
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai8
A Survey on Evaluation of Large Language Models2024
A survey on large language model based autonomous agents2024
Leakage and the reproducibility crisis in machine-learning-based science2023
Evaluating Large Language Models Trained on Code2021
Large Language Models for Software Engineering: A Systematic Literature Review2024
Software Testing With Large Language Models: Survey, Landscape, and Vision2024
The Rise of Agentic AI: A Review of Definitions, Frameworks, Architectures, Applications, Evaluation Metrics, and Challenges2025
Evaluation and Benchmarking of LLM Agents: A Survey2025