Evaluating clinical AI summaries with large language models as judges

Оценка клинических резюме, созданных ИИ, с использованием больших языковых моделей в качестве судей
Matthew M. Churpek, Guanhua Chen, Dmitriy Dligach, Anoop M. Mayampurath, Majid Afshar, Yanjun Gao, Emma Croxford, John Caskey, Brian W. Patterson, Elliot First, Nicholas Pellegrino, Miranda Schnier, Madeline K. Oguss, Graham J Wills, Frank Liao, Cherodeep Goswami, Karen K. Wong
2025-11-05

EHR summarizationLLM-as-a-JudgeProvider Documentation Summarization Quality Instrumentclinical AI summariesinter-rater reliability
Electronic Health Records (EHRs) store vast amounts of clinical information that are difficult for healthcare providers to summarize and synthesize relevant details to their practice. To reduce cognitive load on providers, generative AI with Large Language Models have emerged to automatically summarize patient records into clear, actionable insights and offload the cognitive burden for providers. However, LLM summaries need to be precise and free from errors, making evaluations on the quality of the summaries necessary. While human experts are the gold standard for evaluations, their involvement is time-consuming and costly. Therefore, we introduce and validate an automated method for evaluating real-world EHR multi-document summaries using an LLM as the evaluator, referred to as LLM-as-a-Judge. Benchmarking against the validated Provider Documentation Summarization Quality Instrument (PDSQI)-9 for human evaluation, our LLM-as-a-Judge framework demonstrated strong inter-rater reliability with human evaluators. GPT-o3-mini achieved the highest intraclass correlation coefficient of 0.818 (95% CI 0.772, 0.854), with a median score difference of 0 from human evaluators, and completes evaluations in just 22 seconds. Overall, the reasoning models excelled in inter-rater reliability, particularly in evaluations that require advanced reasoning and domain expertise, outperforming non-reasoning models, those trained on the task, and multi-agent workflows. Cross-task validation on the Problem Summarization task similarly confirmed high reliability. By automating high-quality evaluations, medical LLM-as-a-Judge offers a scalable, efficient solution to rapidly identify accurate and safe AI-generated summaries in healthcare settings.
1
GPT-o3-mini achieved an intraclass correlation coefficient of 0.818 (95% CI 0.772–0.854) and a median score difference of 0 compared with humans.
2
GPT-o3-mini completed summary evaluations in 22 seconds, indicating substantially improved scalability and efficiency over manual review.
3
Reasoning models provided higher inter-rater reliability than non-reasoning, task-trained, and multi-agent approaches, especially for evaluations requiring advanced reasoning and clinical expertise.
4
The LLM-as-a-Judge framework showed strong agreement with human evaluators using the validated Provider Documentation Summarization Quality Instrument.
5
The study introduces and validates an automated large-language-model method for evaluating real-world, multi-document EHR summaries.

AI-generated multi-document clinical summaries of real-world electronic health records (EHRs)

The accuracy, quality, safety, and automated evaluation of EHR clinical summaries using LLMs as judges, including agreement with human evaluators

Publication Details
Publication Date
2025-11-05
Journal
Publisher
ISSN
Cited by
77
Access Type
Author Information
Authors
Matthew M. Churpek
Guanhua Chen
Dmitriy Dligach
Anoop M. Mayampurath
Majid Afshar
Yanjun Gao
Emma Croxford
John Caskey
Brian W. Patterson
Elliot First
Nicholas Pellegrino
Miranda Schnier
Madeline K. Oguss
Graham J Wills
Frank Liao
Cherodeep Goswami
Karen K. Wong
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%