Evaluating the quality of large language model-generated preoperative patient education material: a comparative study across models and surgery types
Оценка качества предоперационных информационных материалов для пациентов, сгенерированных большими языковыми моделями: сравнительное исследование различных моделей и типов хирургических вмешательств
2025-12-11
SCID: 54.1/jxmgcutt
Discuss with AI
PEMAT-PSuitability Assessment of MaterialsVADER sentiment analysislarge language modelspreoperative patient education
Figures from the paper
Abstract (AI)
Background Numerous studies have confirmed the effectiveness of large language models (LLMs) as a patient education tool; however, these studies primarily relied on the method of asking medical questions. So far, no studies have comprehensively assessed the quality of the complete preoperative patient education material (PEM) generated by LLMs from the perspectives of different models and surgical types. Objective This study aims to comprehensively assess and compare the quality of different types of complete preoperative PEM generated by six common LLMs. Design A Cross-sectional Comparative Study. Methods We prompted 6 LLMs to generate preoperative PEMs for 6 distinct surgical types. For each surgical type, the materials were evaluated by 3 groups of experts from relevant fields using a 5-point scale for their accuracy and completeness. Two researchers assessed the materials for understandability and actionability using the PEMAT-P, and for suitability using SAM. We also analyzed the materials for readability with Flesch-Kincaid and for sentiment with the VADER sentiment analysis tool. Statistical analysis was performed using the Friedman test, followed by Conover’s post-hoc test with Bonferroni correction. Results The research results show that each model has its strengths in different dimensions. All the models demonstrated excellent accuracy, understandability, and actionability with no statistically significant differences. In terms of completeness, Grok-4 and Claude-Opus-4 significantly outperformed GPT-4o. For suitability, Claude-Opus-4 performed the best, while Grok-4 was the worst. For readability, Grok-4 and Gemini-2.5-Pro were the easiest to understand, while Claude-Opus-4 had the lowest readability. Moreover, only Gemini-2.5-Pro could consistently generate content with positive emotions. Conclusion The research has found that the materials generated by these models can achieve high levels in multiple dimensions, but there is no perfect model. These models can be used by medical staff to generate the initial draft of preoperative PEMs. However, before providing them to the patients, they still need to be reviewed and supplemented by the medical staff.
Key Findings
1
All models demonstrated excellent accuracy, understandability, and actionability, with no statistically significant differences across these dimensions.
2
Claude-Opus-4 achieved the highest suitability, whereas Grok-4 achieved the lowest suitability among evaluated models.
3
Grok-4 and Claude-Opus-4 significantly outperformed GPT-4o in completeness of preoperative education materials.
4
Grok-4 and Gemini-2.5-Pro produced the most readable materials, Claude-Opus-4 had the lowest readability, and only Gemini-2.5-Pro consistently generated positively emotional content.
5
No single model performed best across every quality dimension, indicating that model selection should depend on the intended educational criterion.
6
The study evaluated complete preoperative patient education materials generated by six large language models across six distinct surgical types.
Research Object
complete preoperative patient education materials generated by six large language models for six surgical types
Research Subject
the quality of the generated materials, including accuracy, completeness, understandability, actionability, suitability, readability, and sentiment, compared across language models and surgical types
Publication Details
Publication Date
2025-12-11
Journal
Publisher
ISSN
Cited by
2
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest