How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment
Как ChatGPT показывает себя на экзамене для получения медицинской лицензии в США (USMLE)? Последствия использования крупных языковых моделей для медицинского образования и оценки знаний
2023-02-08
SCID: 54.1/3rpu3dan
Discuss with AI
ChatGPTNBME-Free-Step-1 datasetUSMLEUnited States Medical Licensing Examinationexplainable answers (logic and informational context)interactive medical education toolknowledge assessmentmedical educationmedical question answeringpassing score (>60%)
Figures from the paper
Abstract (AI)
BACKGROUND: Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user input. OBJECTIVE: This study aimed to evaluate the performance of ChatGPT on questions within the scope of the United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as to analyze responses for user interpretability. METHODS: We used 2 sets of multiple-choice questions to evaluate ChatGPT's performance, each with questions pertaining to Step 1 and Step 2. The first set was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the user base. The second set was the National Board of Medical Examiners (NBME) free 120 questions. ChatGPT's performance was compared to 2 other large language models, GPT-3 and InstructGPT. The text output of each ChatGPT response was evaluated across 3 qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question. RESULTS: Of the 4 data sets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBME-Free-Step2, ChatGPT achieved accuracies of 44% (44/100), 42% (42/100), 64.4% (56/87), and 57.8% (59/102), respectively. ChatGPT outperformed InstructGPT by 8.15% on average across all data sets, and GPT-3 performed similarly to random chance. The model demonstrated a significant decrease in performance as question difficulty increased (P=.01) within the AMBOSS-Step1 data set. We found that logical justification for ChatGPT's answer selection was present in 100% of outputs of the NBME data sets. Internal information to the question was present in 96.8% (183/189) of all questions. The presence of information external to the question was 44.5% and 27% lower for incorrect answers relative to correct answers on the NBME-Free-Step1 (P<.001) and NBME-Free-Step2 (P=.001) data sets, respectively. CONCLUSIONS: ChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at a greater than 60% threshold on the NBME-Free-Step-1 data set, we show that the model achieves the equivalent of a passing score for a third-year medical student. Additionally, we highlight ChatGPT's capacity to provide logic and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as an interactive medical education tool to support learning.
Key Findings
1
ChatGPT achieves greater than 60% on the NBME-Free-Step-1 dataset, equivalent to a passing score for a third-year medical student.
2
ChatGPT frequently provides logical reasoning and informational context across the majority of its answers.
3
ChatGPT shows significant improvement in medical question answering compared to prior natural language processing models.
4
The combination of passing performance and explanatory capacity suggests ChatGPT's potential as an interactive medical education tool to support learning.
Research Object
ChatGPT evaluated on the United States Medical Licensing Examination (USMLE) / NBME-Free-Step-1 question set
Research Subject
Performance on medical question answering including pass-equivalent score (>60%), ability to provide logical reasoning and informational context, and implications for medical education and knowledge assessment
Publication Details
Publication Date
2023-02-08
Journal
Publisher
ISSN
Cited by
2285
Open access PDF
Access Type
Author Information
Download PDF
Subscribe to digest
References available in scid.ai2
Cited by20
A Survey on Evaluation of Large Language Models2024
ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns2023
Practical and ethical challenges of large language models in education: A systematic scoping review2023
ChatGPT for Education and Research: Opportunities, Threats, and Strategies2023
Shaping the Future of Education: Exploring the Potential and Consequences of AI and ChatGPT in Educational Settings2023
The future landscape of large language models in medicine2023
Generative artificial intelligence2023
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage2023
The role of ChatGPT in higher education: Benefits, challenges, and future research directions2023
A scoping review of artificial intelligence in medical education: BEME Guide No. 842024
Generative AI in healthcare: an implementation science informed translational path on application, integration and governance2024
A framework for human evaluation of large language models in healthcare derived from literature review2024
Improving large language models for clinical named entity recognition via prompt engineering2023
Large language models in medical and healthcare fields: applications, advances, and challenges2024
Prompt Engineering in Medical Education2023
Opportunities, challenges, and future directions of large language models, including ChatGPT in medical education: a systematic scoping review2024
HuntGPT: Integrating Machine Learning-Based Anomaly Detection and Explainable AI with Large Language Models (LLMs)2026
Using ChatGPT for human–computer interaction research: a primer2023
A large language model digital patient system enhances ophthalmology history taking skills2025
Large language model answers medical questions about standard pathology reports2024