MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents
MedAgentBench: виртуальная среда электронной медицинской карты для тестирования медицинских агентов на основе больших языковых моделей
2025-08-14
SCID: 54.1/99fesyjh
Discuss with AI
FHIR-compliant environmentMedAgentBenchagent-oriented benchmarkelectronic health recordsmedical LLM agents
Figures from the paper
Abstract (AI)
BACKGROUND: Recent large language models (LLMs) have demonstrated significant advancements, particularly in their ability to serve as agents, thereby surpassing their traditional role as chatbots. These agents can leverage their planning and tool utilization capabilities to address tasks specified at a high level. This suggests new potential to reduce the burden of administrative tasks and address current health care staff shortages. However, a standardized dataset to benchmark the agent capabilities of LLMs in medical applications is currently lacking, making it difficult to evaluate their performance on complex tasks in interactive health care environments. METHODS: To address this gap in the deployment of agentic artificial intelligence (AI) in health care, we introduce MedAgentBench, a broad evaluation suite designed to assess the agent capabilities of LLMs within medical records contexts. MedAgentBench encompasses 300 patient-specific clinically derived tasks from 10 categories written by human physicians, realistic profiles of 100 patients with over 700,000 data elements, a Fast Healthcare Interoperability Resources-compliant interactive environment, and an accompanying codebase. The environment uses standard application programming interfaces and communication infrastructure used in modern electronic health record (EHR) systems so that it can be easily migrated into live EHR systems. RESULTS: MedAgentBench presents an unsaturated agent-oriented benchmark at which current state-of-the-art LLMs exhibit some ability to succeed. The best model (Claude 3.5 Sonnet v2) achieves a success rate of 69.67%. However, there is still substantial room for improvement, which gives the community a clear direction for future optimization efforts. Furthermore, there is significant variation in performance across task categories. CONCLUSIONS: Agent-based task frameworks and benchmarks are the necessary next step to advance the potential and capabilities for effectively improving and integrating AI systems into clinical workflows. MedAgentBench establishes this and is publicly available at https://github.com/stanfordmlgroup/MedAgentBench, offering a valuable framework for model developers to track progress and drive continuous improvements in the agent capabilities of LLMs within the medical domain. (Funded by the NIH and Singapore's National Science Scholarship [PhD].).
Key Findings
1
LLM-agent performance varies significantly across task categories, highlighting uneven capabilities in medical EHR interactions.
2
MedAgentBench introduces a standardized benchmark for evaluating medical LLM agents on complex, interactive electronic health record tasks.
3
MedAgentBench provides a FHIR-compliant interactive environment using standard EHR APIs and communication infrastructure, supporting potential migration into live systems.
4
The benchmark contains 300 physician-written, patient-specific tasks across 10 categories, realistic profiles for 100 patients, and more than 700,000 data elements.
5
The benchmark remains unsaturated: Claude 3.5 Sonnet v2 achieves the highest reported success rate at 69.67%, indicating substantial room for improvement.
Research Object
MedAgentBench’s virtual, FHIR-compliant interactive electronic health record environment containing realistic patient profiles and clinically derived tasks
Research Subject
The task-solving capabilities and performance of medical LLM agents in interactive EHR contexts, including variation across task categories
Publication Details
Publication Date
2025-08-14
Journal
Publisher
ISSN
Cited by
66
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai2
Cited by6
Towards autonomous medical artificial intelligence agents2026
A comprehensive survey of AI agents in healthcare2026
Agentic AI in Healthcare and Medicine: A Seven-Dimensional Taxonomy for Empirical Evaluation of LLM-Based Agents2026
PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents2026
Artificial intelligence agent for delirium screening among patients in oncology and cardiac intensive care units: a proof-of-concept study2026
AgentInspect: Diagnosing Behavioral Failures in Artificial Intelligence Agents2026