ASN's Mission

To create a world without kidney diseases, the ASN Alliance for Kidney Health elevates care by educating and informing, driving breakthroughs and innovation, and advocating for policies that create transformative changes in kidney medicine throughout the world.

learn more

Contact ASN

1401 H St, NW, Ste 900, Washington, DC 20005

email@asn-online.org

202-640-4660

The Latest on X

Kidney Week

Abstract: TH-PO009

Artificial Intelligence Language Processing Models in Literature Reviews for Nephrology

Session Information

Category: Augmented Intelligence, Digital Health, and Data Science

  • 300 Augmented Intelligence, Digital Health, and Data Science

Authors

  • Suppadungsuk, Supawadee, Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Krisanapan, Pajaree, Mayo Clinic Minnesota, Rochester, United States
  • Tangpanithandee, Supawit, Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Garcia Valencia, Oscar Alejandro, Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Thongprayoon, Charat, Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Kashani, Kianoush, Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Cheungpasitporn, Wisit, Mayo Clinic Minnesota, Rochester, Minnesota, United States
Background

Literature reviews are a valuable tool for summarizing and evaluating the available evidence in various medical fields, including Nephrology. However, identifying and selecting relevant literature can be time-consuming for clinicians and researchers. ChatGPT is a novel artificial intelligence (AI) language model renowned for its exceptional ability to generate human-like responses across various tasks. However, whether ChatGPT can effectively assist medical professionals in identifying relevant literature is unclear. Therefore, this study aimed to assess the effectiveness of ChatGPT in identifying references to literature reviews in Nephrology.

Methods

We keyed the prompt "What are the references to literature reviews in Nephrology, with subgroup analysis in specific areas?" into ChatGPT (03/23 Version). We selected all provided results by ChatGPT and assessed them for existence, relevance, and author/link correctness. We recorded each resource's citations, references, authors' names, and links. The relevance of each resource was assessed by its citation index on Google Scholar.

Results

Of the total 511 references in Nephrology literature, only 319 (62.43%) of the references provided by ChatGPT existed, while 30.3% did not exist, and in 7.3% of recommendations, they were incomplete. Additionally, 208 (40.7%) of the listed authors were found to be incorrect, and the digital object identifier (DOI) was inaccurate in 357 (69.86%) of the references. Moreover, among those with a link provided, the link was correct in only 108 (21.14%) cases, and 7.4% of the references were irrelevant. Notably, analysis of specific topics in electrolyte, hemodialysis, and kidney stones found that >60% of the references were inaccurate or misleading, with less reliable authors and links provided by ChatGPT.

Conclusion

Based on our findings, the use of ChatGPT as a sole resource for identifying references to literature reviews in Nephrology is not recommended. Future studies could explore ways to improve AI language models' performance in identifying relevant Nephrology literature.