MAIA: A Multidimensional Benchmark for Assessing Medical AI Agents

Large language models show remarkable potential in medical scenarios, especially as autonomous agents for complex clinical reasoning. Rigorous evaluation is essential to ensure their reliability in real-world healthcare applications. However, existing medical benchmarks suffer from narrow task scopes, dependence on public datasets prone to data leakage, and limited coverage of diverse agent capabilities. To address these gaps, we introduce Medical AI Assessment (MAIA), a comprehensive benchmark evaluating medical agents along three dimensions: retrieval-based medical questions generated through biomedical APIs, multi-hop reasoning tasks derived from curated biomedical knowledge graphs, clinical-pathway reasoning questions constructed from authoritative guidelines. MAIA leverages large language models for automatic question generation, reducing manual effort while maintaining clinical fidelity and reasoning depth. Experiments across base and reasoning models reveal both strengths and gaps, underscoring MAIA’s value for advancing medical agent evaluation. MAIA is publicly available at https://huggingface.co/datasets/DiligentDing/MAIA.