Red teaming ChatGPT in medicine to yield real-world insights on model behavior

Проверка ChatGPT методом red teaming в медицине для получения практических сведений о поведении модели
Yanan Ding, Archer Yi Yang, Jonathan H. Chen, Lü Tian, Matthew Schwede, Nigam Haresh Shah, Jesutofunmi A. Omiye, Haiwen Gui, Shawheen J. Rezaei, Roxana Daneshjou, Scott L. Fleming, Jason Fries, Balasubramanian Narasimhan, Juan M. Banda, Crystal T. Chang, Hodan Farah, Charbel Bou-Khalil, Ye-Jean Park, Akshay Swaminathan, Akaash Kolluri, Akash Chaurasia, Alejandro Lozano, Alice Heiman, Allison Sihan Jia, Amit Kaushal, Angela Y. Jia, Angelica Iacovelli, Arghavan Salles, Arpita Singhal, Benjamin Belai, Benjamin H. Jacobson, Binglan Li, Celeste H. Poe, Chandan Sanghera, Chenming Zheng, Conor Messer, Damien Varid Kettud, Deven Pandya, Dhamanpreet Kaur, Diana A. Hla, Diba Dindoust, Dominik Moehrle, Ross Duncan, Ellaine Chou, Eric C. Lin, Fateme Nateghi Haredasht, Cheng Ge, Irena Gao, Jacob Chang, Jake Silberg, Jiapeng Xu, J. Weston Jamison, John S. Tamaresis, Joshua Lazaro, Julie Lee, Karen Ebert Matthys, Kirsten R. Steffner, Luca Pegolotti, Malathi Srinivasan, Maniragav Manimaran, Minghe Zhang, Minh Hoai Nguyen, Mohsen Fathzadeh, Qian Zhao, Rika Bajra, Rohit Khurana, Ruhana Azam, R. W. Bartlett, Sang Truong, S. Varadha Raj, Solveig Behr, Sonia Onyeka, Sri Muppidi, Tarek Bandali, Tiffany Y. Eulalio, Wenyuan Chen, Xuanyu Zhou, Ying Cui, Yuqi Tan, Yutong Liu, Arghavan Salles, Yanan Ding, Qian Zhao, Ye‐Jean Park, Minh Nguyen, Shriti Raj, Angela Jia, Conor Messer, Jake Silberg, Joshua E. Lazaro, Rohit Khurana, Ruhana Azam, Rush Bartlett, Tarek Bandali, Xuanyu Zhou
2025-03-07

clinical caseshallucinations and biaslarge language modelsmodel safetyred teaming
0. Abstract Background The integration of large language models (LLMs) in healthcare offers immense opportunity to streamline healthcare tasks, but also carries risks such as response accuracy and bias perpetration. To address this, we conducted a red-teaming exercise to assess LLMs in healthcare and developed a dataset of clinically relevant scenarios for future teams to use. Methods We convened 80 multi-disciplinary experts to evaluate the performance of popular LLMs across multiple medical scenarios. Teams composed of clinicians, medical and engineering students, and technical professionals stress-tested LLMs with real world clinical use cases. Teams were given a framework comprising four categories to analyze for inappropriate responses: Safety, Privacy, Hallucinations, and Bias. Prompts were tested on GPT-3.5, GPT-4.0, and GPT-4.0 with the Internet. Six medically trained reviewers subsequently reanalyzed the prompt-response pairs, with dual reviewers for each prompt and a third to resolve discrepancies. This process allowed for the accurate identification and categorization of inappropriate or inaccurate content within the responses. Results There were a total of 382 unique prompts, with 1146 total responses across three iterations of ChatGPT (GPT-3.5, GPT-4.0, GPT-4.0 with Internet). 19.8% of the responses were labeled as inappropriate, with GPT-3.5 accounting for the highest percentage at 25.7% while GPT-4.0 and GPT-4.0 with internet performing comparably at 16.2% and 17.5% respectively. Interestingly, 11.8% of responses were deemed appropriate with GPT-3.5 but inappropriate in updated models, highlighting the ongoing need to evaluate evolving LLMs. Conclusion The red-teaming exercise underscored the benefits of interdisciplinary efforts, as this collaborative model fosters a deeper understanding of the potential limitations of LLMs in healthcare and sets a precedent for future red teaming events in the field. Additionally, we present all prompts and outputs as a benchmark for future LLM model evaluations. 1-2 Sentence Description As a proof-of-concept, we convened an interactive “red teaming” workshop in which medical and technical professionals stress-tested popular large language models (LLMs) through publicly available user interfaces on clinically relevant scenarios. Results demonstrate a significant proportion of inappropriate responses across GPT-3.5, GPT-4.0, and GPT-4.0 with Internet (25.7%, 16.2%, and 17.5%, respectively) and illustrate the valuable role that non-technical clinicians can play in evaluating models.
1
A non-model-creator-affiliated red-teaming effort engaged 80 clinicians, students, and technical professionals to evaluate medical LLM behavior using real-world clinical cases.
2
Among 376 unique prompts generating 1,504 responses, 20.1% were inappropriate across safety, privacy, hallucination/accuracy, and bias dimensions.
3
Inappropriate-response rates differed by model: 25.8% for GPT-3.5, 16% for GPT-4.0, and 17.8% for GPT-4.0 with Internet access.
4
Model updates did not consistently preserve safe behavior: 21.5% of responses appropriate with GPT-3.5 became inappropriate in updated models.
5
The benchmark detected 20.4% inappropriate responses in GPT-4o, despite its release after the original red-teaming event.

ChatGPT and related GPT models used for real-world clinical cases

Model behavior, including safety, privacy, hallucinations/accuracy, bias, and changes in inappropriate-response rates under adversarial clinical testing

Publication Details
Publication Date
2025-03-07
Journal
Publisher
ISSN
Cited by
51
Access Type
Author Information
Authors
Yanan Ding
Archer Yi Yang
Jonathan H. Chen
Lü Tian
Matthew Schwede
Nigam Haresh Shah
Jesutofunmi A. Omiye
Haiwen Gui
Shawheen J. Rezaei
Roxana Daneshjou
Scott L. Fleming
Jason Fries
Balasubramanian Narasimhan
Juan M. Banda
Crystal T. Chang
Hodan Farah
Charbel Bou-Khalil
Ye-Jean Park
Akshay Swaminathan
Akaash Kolluri
Akash Chaurasia
Alejandro Lozano
Alice Heiman
Allison Sihan Jia
Amit Kaushal
Angela Y. Jia
Angelica Iacovelli
Arghavan Salles
Arpita Singhal
Benjamin Belai
Benjamin H. Jacobson
Binglan Li
Celeste H. Poe
Chandan Sanghera
Chenming Zheng
Conor Messer
Damien Varid Kettud
Deven Pandya
Dhamanpreet Kaur
Diana A. Hla
Diba Dindoust
Dominik Moehrle
Ross Duncan
Ellaine Chou
Eric C. Lin
Fateme Nateghi Haredasht
Cheng Ge
Irena Gao
Jacob Chang
Jake Silberg
Jiapeng Xu
J. Weston Jamison
John S. Tamaresis
Joshua Lazaro
Julie Lee
Karen Ebert Matthys
Kirsten R. Steffner
Luca Pegolotti
Malathi Srinivasan
Maniragav Manimaran
Minghe Zhang
Minh Hoai Nguyen
Mohsen Fathzadeh
Qian Zhao
Rika Bajra
Rohit Khurana
Ruhana Azam
R. W. Bartlett
Sang Truong
S. Varadha Raj
Solveig Behr
Sonia Onyeka
Sri Muppidi
Tarek Bandali
Tiffany Y. Eulalio
Wenyuan Chen
Xuanyu Zhou
Ying Cui
Yuqi Tan
Yutong Liu
Arghavan Salles
Yanan Ding
Qian Zhao
Ye‐Jean Park
Minh Nguyen
Shriti Raj
Angela Jia
Conor Messer
Jake Silberg
Joshua E. Lazaro
Rohit Khurana
Ruhana Azam
Rush Bartlett
Tarek Bandali
Xuanyu Zhou
Explore further
Open the scid.ai AI chat with a ready-made request: it will find papers on a similar topic and help build a literature review.
Find similar papers in the chat →
Make a presentation
100%