Abstract: TH-PO004
Unraveling ChatGPT's Performance in Addressing ESKD: Implications for Artificial Intelligence (AI)-Assisted Healthcare
Session Information
- AI, Digital Health, Data Science - I
November 02, 2023 | Location: Exhibit Hall, Pennsylvania Convention Center
Abstract Time: 10:00 AM - 12:00 PM
Category: Augmented Intelligence, Digital Health, and Data Science
- 300 Augmented Intelligence, Digital Health, and Data Science
Authors
- Davis, Paul W., Mayo Clinic Rochester, Rochester, Minnesota, United States
- Garcia Valencia, Oscar Alejandro, Mayo Clinic Rochester, Rochester, Minnesota, United States
- Craici, Iasmina, Mayo Clinic Rochester, Rochester, Minnesota, United States
- Kattah, Andrea G., Mayo Clinic Rochester, Rochester, Minnesota, United States
- Thongprayoon, Charat, Mayo Clinic Rochester, Rochester, Minnesota, United States
- Gregoire, James Robert, Mayo Clinic Rochester, Rochester, Minnesota, United States
- Dillon, John J., Mayo Clinic Rochester, Rochester, Minnesota, United States
- Cheungpasitporn, Wisit, Mayo Clinic Rochester, Rochester, Minnesota, United States
Background
ChatGPT, an artificial intelligence language model, is at the forefront of cutting-edge technology. It has shown abilities in natural language processing tasks, producing responses resembling those crafted by human beings. While there is discourse about the potential of ChatGPT as a substitute for physicians, its abilities in the field of nephrology, particularly in ESKD including dialysis, remains uncertain. The objective of this study is to assess the performance of ChatGPT in addressing fundamental inquiries pertaining to ESKD.
Methods
We conducted an evaluation of ChatGPT's accuracy in answering questions related to CKD, ESKD, including hemodialysis, and peritoneal dialysis, using the ASN eLEARNING CENTER (nephSAP vol1-No2 and Dialysis Core Curriculum 2021). There were 95 questions included. Each question set was executed twice using ChatGPT (Mar 14 version, OpenAI), and the level of agreement between the initial and subsequent run, conducted two weeks apart, was determined. Also, an assessment was performed using ChatGPT using the query, "Based on these findings, what is ChatGPT's performance, and is ChatGPT ready to provide answers pertaining to ESKD?"
Results
In our study evaluating ChatGPT's performance in answering questions related to CKD and ESKD, we found that on the two different question banks combined, ChatGPT achieved accuracies of 54% and 57% on the first and second runs, respectively. The overall agreement between the two runs was 71%. The study revealed that the level of agreement between the initial and subsequent runs of ChatGPT was higher for correct answers compared to incorrect ones, concordance of 46% vs 24%, respectively. Among the 28 instances where ChatGPT provided different responses, it changed from incorrect to correct in 10 questions (36%), from correct to incorrect 7 times (25%). ChatGPT acknowledged these results, further highlighting its limitations in accurately addressing questions related to ESKD.
Conclusion
The current study demonstrates that ChatGPT's accuracy in answering questions related to ESKD is below the minimum passing threshold of 75% set by the ASN for nephrologists, with an accuracy of 55% (average of the two runs), indicating the need for further development and training to improve its accuracy and consistency.