Evaluating clinical AI summaries with large language models as judges
Оценка клинических резюме, созданных ИИ, с использованием больших языковых моделей в качестве судей
2025-11-05
SCID: 54.1/suuntcju
Discuss with AI
EHR summarizationLLM-as-a-JudgeProvider Documentation Summarization Quality Instrumentclinical AI summariesinter-rater reliability
Figures from the paper
Abstract (AI)
Electronic Health Records (EHRs) store vast amounts of clinical information that are difficult for healthcare providers to summarize and synthesize relevant details to their practice. To reduce cognitive load on providers, generative AI with Large Language Models have emerged to automatically summarize patient records into clear, actionable insights and offload the cognitive burden for providers. However, LLM summaries need to be precise and free from errors, making evaluations on the quality of the summaries necessary. While human experts are the gold standard for evaluations, their involvement is time-consuming and costly. Therefore, we introduce and validate an automated method for evaluating real-world EHR multi-document summaries using an LLM as the evaluator, referred to as LLM-as-a-Judge. Benchmarking against the validated Provider Documentation Summarization Quality Instrument (PDSQI)-9 for human evaluation, our LLM-as-a-Judge framework demonstrated strong inter-rater reliability with human evaluators. GPT-o3-mini achieved the highest intraclass correlation coefficient of 0.818 (95% CI 0.772, 0.854), with a median score difference of 0 from human evaluators, and completes evaluations in just 22 seconds. Overall, the reasoning models excelled in inter-rater reliability, particularly in evaluations that require advanced reasoning and domain expertise, outperforming non-reasoning models, those trained on the task, and multi-agent workflows. Cross-task validation on the Problem Summarization task similarly confirmed high reliability. By automating high-quality evaluations, medical LLM-as-a-Judge offers a scalable, efficient solution to rapidly identify accurate and safe AI-generated summaries in healthcare settings.
Key Findings
1
GPT-o3-mini achieved an intraclass correlation coefficient of 0.818 (95% CI 0.772–0.854) and a median score difference of 0 compared with humans.
2
GPT-o3-mini completed summary evaluations in 22 seconds, indicating substantially improved scalability and efficiency over manual review.
3
Reasoning models provided higher inter-rater reliability than non-reasoning, task-trained, and multi-agent approaches, especially for evaluations requiring advanced reasoning and clinical expertise.
4
The LLM-as-a-Judge framework showed strong agreement with human evaluators using the validated Provider Documentation Summarization Quality Instrument.
5
The study introduces and validates an automated large-language-model method for evaluating real-world, multi-document EHR summaries.
Research Object
AI-generated multi-document clinical summaries of real-world electronic health records (EHRs)
Research Subject
The accuracy, quality, safety, and automated evaluation of EHR clinical summaries using LLMs as judges, including agreement with human evaluators
Publication Details
Publication Date
2025-11-05
Journal
Publisher
ISSN
Cited by
77
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest