Evaluating Artificial Intelligence in Patient Education: DeepSeek ‐ V3 Versus ChatGPT ‐4o in Answering Common Questions on Laparoscopic Cholecystectomy
Оценка искусственного интеллекта в обучении пациентов: DeepSeek-V3 и ChatGPT-4o в ответах на распространённые вопросы о лапароскопической холецистэктомии
2025-06-11
SCID: 54.1/dgdqpvq5
Discuss with AI
ChatGPT-4oDeepSeek-V3laparoscopic cholecystectomylarge language modelspatient education
Figures from the paper
Abstract (AI)
BACKGROUND: Artificial intelligence-based large language models (AI-based LLMs) have gained popularity over traditional search engines for obtaining medical information. However, the accuracy and reliability of these AI-generated medical insights remain a topic of debate. Recently, a new AI-based LLM, DeepSeek-V3, developed in East Asia, has been introduced. The aim of this study is to evaluate the appropriateness, accuracy, and readability of responses and the usability of these answers for patient education provided by ChatGPT-4o and DeepSeek-V3 AI-based LLMs to frequently asked questions by patients regarding laparoscopic cholecystectomy (LC). METHODS: The 20 most frequently asked questions by patients regarding LC were presented to the DeepSeek-V3 and ChatGPT-4o chatbots. Before each question, the search history was deleted. The comprehensiveness of the responses was evaluated based on clinical experience by two board-certified general surgeons experienced in hepatobiliary surgery using a Likert scale. Paired sample t-test and Wilcoxon signed rank test were used. Inter-rater reliability was analyzed with Cohen's Kappa test. RESULTS: The DeepSeek-V3 chatbot provided statistically significantly more suitable responses compared to ChatGPT-4o (p = 0.033). On the Likert scale, DeepSeek-V3 received a 5-point rating for 19 out of 20 questions (95%), whereas ChatGPT-4o achieved a 5-point rating for only 13 questions (65%). Based on the evaluation conducted according to the reviewers' clinical experience, DeepSeek-V3 provided statistically significantly more appropriate responses (p = 0.008). CONCLUSION: Released in January 2025, DeepSeek-V3 provides more suitable responses to patient inquiries regarding LC compared to ChatGPT-4o.
Key Findings
1
Clinical reviewers again found DeepSeek-V3 responses significantly more appropriate than ChatGPT-4o responses (p = 0.008).
2
DeepSeek-V3 produced statistically significantly more suitable responses than ChatGPT-4o according to surgeon evaluations (p = 0.033).
3
DeepSeek-V3 received the highest Likert rating for 19 of 20 questions (95%), compared with 13 of 20 questions (65%) for ChatGPT-4o.
4
The findings support DeepSeek-V3 as a potentially more suitable AI tool than ChatGPT-4o for patient education about laparoscopic cholecystectomy.
5
The study evaluated DeepSeek-V3 and ChatGPT-4o responses to 20 frequently asked patient questions about laparoscopic cholecystectomy.
Research Object
DeepSeek-V3 and ChatGPT-4o responses to patients’ frequently asked questions about laparoscopic cholecystectomy
Research Subject
Comparative appropriateness, accuracy, comprehensiveness, readability, and usability of AI-generated answers for patient education
Publication Details
Publication Date
2025-06-11
Journal
Publisher
ISSN
Cited by
20
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest