ASN's Mission

To create a world without kidney diseases, the ASN Alliance for Kidney Health elevates care by educating and informing, driving breakthroughs and innovation, and advocating for policies that create transformative changes in kidney medicine throughout the world.

learn more

Contact ASN

1401 H St, NW, Ste 900, Washington, DC 20005

email@asn-online.org

202-640-4660

The Latest on X

Kidney Week

Abstract: TH-PO004

Unraveling ChatGPT's Performance in Addressing ESKD: Implications for Artificial Intelligence (AI)-Assisted Healthcare

Session Information

Category: Augmented Intelligence, Digital Health, and Data Science

  • 300 Augmented Intelligence, Digital Health, and Data Science

Authors

  • Davis, Paul W., Mayo Clinic Rochester, Rochester, Minnesota, United States
  • Garcia Valencia, Oscar Alejandro, Mayo Clinic Rochester, Rochester, Minnesota, United States
  • Craici, Iasmina, Mayo Clinic Rochester, Rochester, Minnesota, United States
  • Kattah, Andrea G., Mayo Clinic Rochester, Rochester, Minnesota, United States
  • Thongprayoon, Charat, Mayo Clinic Rochester, Rochester, Minnesota, United States
  • Gregoire, James Robert, Mayo Clinic Rochester, Rochester, Minnesota, United States
  • Dillon, John J., Mayo Clinic Rochester, Rochester, Minnesota, United States
  • Cheungpasitporn, Wisit, Mayo Clinic Rochester, Rochester, Minnesota, United States
Background

ChatGPT, an artificial intelligence language model, is at the forefront of cutting-edge technology. It has shown abilities in natural language processing tasks, producing responses resembling those crafted by human beings. While there is discourse about the potential of ChatGPT as a substitute for physicians, its abilities in the field of nephrology, particularly in ESKD including dialysis, remains uncertain. The objective of this study is to assess the performance of ChatGPT in addressing fundamental inquiries pertaining to ESKD.

Methods

We conducted an evaluation of ChatGPT's accuracy in answering questions related to CKD, ESKD, including hemodialysis, and peritoneal dialysis, using the ASN eLEARNING CENTER (nephSAP vol1-No2 and Dialysis Core Curriculum 2021). There were 95 questions included. Each question set was executed twice using ChatGPT (Mar 14 version, OpenAI), and the level of agreement between the initial and subsequent run, conducted two weeks apart, was determined. Also, an assessment was performed using ChatGPT using the query, "Based on these findings, what is ChatGPT's performance, and is ChatGPT ready to provide answers pertaining to ESKD?"

Results

In our study evaluating ChatGPT's performance in answering questions related to CKD and ESKD, we found that on the two different question banks combined, ChatGPT achieved accuracies of 54% and 57% on the first and second runs, respectively. The overall agreement between the two runs was 71%. The study revealed that the level of agreement between the initial and subsequent runs of ChatGPT was higher for correct answers compared to incorrect ones, concordance of 46% vs 24%, respectively. Among the 28 instances where ChatGPT provided different responses, it changed from incorrect to correct in 10 questions (36%), from correct to incorrect 7 times (25%). ChatGPT acknowledged these results, further highlighting its limitations in accurately addressing questions related to ESKD.

Conclusion

The current study demonstrates that ChatGPT's accuracy in answering questions related to ESKD is below the minimum passing threshold of 75% set by the ASN for nephrologists, with an accuracy of 55% (average of the two runs), indicating the need for further development and training to improve its accuracy and consistency.