AgentClinic: a multimodal benchmark for tool-using clinical AI agents
AgentClinic: мультимодальный бенчмарк для клинических ИИ-агентов, использующих инструменты
2026-04-27
SCID: 54.1/a89b5wv2
Discuss with AI
clinical simulation benchmarkelectronic health recordsmultimodal clinical AI agentssequential decision-makingtool use
Figures from the paper
Abstract (AI)
Evaluating large language models (LLM) in clinical scenarios is crucial to assessing their potential clinical utility. Existing benchmarks rely heavily on static question-answering, which does not accurately depict the complex, sequential nature of clinical decision-making. Here, we introduce AgentClinic, a multimodal agent benchmark for evaluating LLMs in simulated clinical environments that include patient interactions, multimodal data collection under incomplete information, and the usage of various tools, resulting in an in-depth evaluation across nine medical specialties and seven languages. We find that solving MedQA problems in the sequential decision-making format of AgentClinic is considerably more challenging, resulting in diagnostic accuracies that can drop to below a tenth of the original accuracy. Overall, we observe that agents sourced from Claude-3.5 outperform other LLM backbones in most settings. Nevertheless, we see stark differences in the LLMs' ability to make use of tools, such as experiential learning, adaptive retrieval, and reflection cycles. Strikingly, Llama-3 shows up to 92% relative improvements with the notebook tool that allows for writing and editing notes that persist across cases. To further scrutinize our clinical simulations, we leverage real-world electronic health records, perform a clinical reader study, perturb agents with biases, and explore patient-centric metrics that this interactive environment enables.
Key Findings
1
AgentClinic introduces a multimodal benchmark for tool-using clinical agents in simulated environments with patient interaction, incomplete information, and sequential tool use.
2
Claude-3.5-based agents outperform other evaluated LLM backbones in most settings, while models differ markedly in their ability to use experiential learning, adaptive retrieval, and reflection tools.
3
Llama-3 achieves up to 92% relative improvement when using a persistent notebook tool for writing and editing notes across cases.
4
Recasting MedQA as sequential decision-making substantially increases difficulty, with diagnostic accuracy dropping below one-tenth of the original accuracy.
5
The benchmark evaluates clinical agents across nine medical specialties and seven languages, extending beyond static question-answering tasks.
6
The benchmark is further examined using real-world electronic health records, clinical reader studies, bias perturbations, and patient-centric metrics enabled by interactive simulations.
Research Object
LLM-based clinical AI agents operating in simulated multimodal clinical environments
Research Subject
their sequential clinical decision-making, diagnostic accuracy, tool use, and patient-centric performance under incomplete information
Publication Details
Publication Date
2026-04-27
Journal
Publisher
ISSN
Cited by
369
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai8
LLaMA: Open and Efficient Foundation Language Models2023
Retrieval-Augmented Generation for Large Language Models: A Survey2023
Large language models encode clinical knowledge2023
MIMIC-IV, a freely accessible electronic health record dataset2023
A survey on large language model based autonomous agents2024
Generative Agents: Interactive Simulacra of Human Behavior2023
Active Retrieval Augmented Generation2023
Benchmarking Retrieval-Augmented Generation for Medicine2024
Cited by11
From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review2026
Reliability of LLMs as medical assistants for the general public: a randomized preregistered study2026
MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents2025
Towards autonomous medical artificial intelligence agents2026
A comprehensive survey of AI agents in healthcare2026
Agentic AI in Healthcare and Medicine: A Seven-Dimensional Taxonomy for Empirical Evaluation of LLM-Based Agents2026
AI agent in healthcare: applications, evaluations, and future directions2026
Agentic AI in Healthcare: Opportunities, Challenges, and Future Directions2026
The Hitchhiker’s Guide to Autonomous Research: A Survey of Scientific Agents2026
PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents2026
Enhancing Psychiatry Training Using an Agentic AI Simulated Consultation Tool: Prospective Cohort Study2026