From benchmarks to deployment: a comprehensive review of agentic AI evaluation

От бенчмарков к внедрению: всесторонний обзор оценки агентного искусственного интеллекта
Adnan Akhunzada, Nikola Ivković, Ahmad Sami Al‐Shamayleh, Tanzila Kehkashan, Muhammad Irsyad Abdullah, Nor Azman Ismail, Sharifah Sakinah Syed Ahmad, Abdul Rehman
2026-04-24

agent benchmarksagentic AI evaluationdeployment viabilitysafety-aware scoringtrajectory-level evaluation
This review systematically examines evaluation methodologies for agentic AI systems, agentic AI systems capable of multi-step planning, tool usage, and environmental interaction across diverse domains. Current evaluation practices exhibit a critical disconnect between benchmark performance and deployment viability, where agents achieving high scores on standardized benchmarks frequently fail in real world applications due to fundamental inadequacies in assessment methodologies that prioritize task completion over deployment critical dimensions such as cost efficiency, safety compliance, maintainability, and workflow integration. We critically analyze 15 major agent benchmarks including AgentBench, WebArena, SWE-bench, PaperBench, MLGym, BrowserGym, HumanEval, MBPP, GAIA, ToolBench, Terminal-Bench, Mind2Web, ALFWorld, BabyAI, and HotPotQA, examining their methodologies, metrics, datasets, and application domains with software development serving as our primary case study. Our analysis reveals that evaluation methodology not model capability constitutes the primary bottleneck limiting reliable agent deployment. We demonstrate how test passing metrics systematically ignore code quality, security vulnerabilities, and integration complexity while binary success metrics obscure planning coherence, resource efficiency, and safety violations (0/15 benchmarks integrate security or safety into scoring. This review provides a cross-domain taxonomy exposing evaluation inadequacies, trajectory level evaluation frameworks addressing cost reproducibility validity trade-offs, systematic identification of metric insufficiencies including absence of safety-aware and cost aware scoring, and synthesis of emerging evaluation paradigms with adoption barriers. Quantitatively, 0/15 benchmarks integrate safety or security into scoring, 0/15 include cost-efficiency metrics in their primary evaluation protocol, and 13/15 rely exclusively on binary success measures confirming that evaluation methodology, not model capability, is the primary bottleneck to reliable deployment. Progress toward trustworthy agentic AI fundamentally depends on evolving evaluation infrastructure beyond binary metrics toward comprehensive, multidimensional assessment frameworks that capture these deployment-essential dimensions.
1
Across 15 major benchmarks, 0/15 integrate safety or security into scoring, 0/15 include cost-efficiency metrics, and 13/15 rely exclusively on binary success measures.
2
Binary task-success and test-passing metrics systematically overlook planning coherence, resource efficiency, code quality, security vulnerabilities, maintainability, and integration complexity.
3
Current agentic AI benchmarks show a critical disconnect between benchmark performance and deployment viability because they prioritize task completion over operational requirements.
4
The review identifies evaluation methodology—not model capability—as the primary bottleneck limiting reliable deployment of agentic AI systems.
5
The review proposes a cross-domain taxonomy and trajectory-level evaluation frameworks addressing cost, reproducibility, validity, safety-aware scoring, and cost-aware scoring, while identifying adoption barriers.

agentic AI systems capable of multi-step planning, tool usage, and environmental interaction

evaluation methodologies and metrics for deployment viability, including task performance, cost efficiency, safety compliance, maintainability, and workflow integration

Publication Details
Publication Date
2026-04-24
Journal
Publisher
ISSN
Cited by
16
Access Type
Author Information
Authors
Adnan Akhunzada
Nikola Ivković
Ahmad Sami Al‐Shamayleh
Tanzila Kehkashan
Muhammad Irsyad Abdullah
Nor Azman Ismail
Sharifah Sakinah Syed Ahmad
Abdul Rehman
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%