ASN's Mission

To create a world without kidney diseases, the ASN Alliance for Kidney Health elevates care by educating and informing, driving breakthroughs and innovation, and advocating for policies that create transformative changes in kidney medicine throughout the world.

learn more

Contact ASN

1401 H St, NW, Ste 900, Washington, DC 20005

email@asn-online.org

202-640-4660

The Latest on X

Kidney Week

Abstract: TH-PO007

The Accuracy of Artificial Intelligence in Identifying Potentially Harmful Non-Prescription Medications and Dietary Supplements in Patients with Kidney Diseases

Session Information

Category: Augmented Intelligence, Digital Health, and Data Science

  • 300 Augmented Intelligence, Digital Health, and Data Science

Authors

  • Sheikh, M. Salman, Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Barreto, Erin F., Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Dreesman, Benjamin, Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Qureshi, Fawad, Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Gregoire, James Robert, Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Erickson, Stephen B., Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Thongprayoon, Charat, Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Craici, Iasmina, Mayo Clinic Minnesota, Rochester, Minnesota, United States
  • Cheungpasitporn, Wisit, Mayo Clinic Minnesota, Rochester, Minnesota, United States
Background

Search engines are commonly used to obtain health-related information, including information about drug safety. Non-prescription medications and supplements are generally perceived as safe, however, may be harmful to patients with kidney diseases. ChatGPT is a cutting-edge language model that has gained attention for its potential to improve clinical decision making. Its ability to serve as a drug information resource for patients with kidney disease has not been determined. This study aimed to evaluate ChatGPT’s accuracy in discerning the safety of medications in patients with kidney diseases when compared to Micromedex, a widely used tertiary drug information reference.

Methods

One hundred twenty-four commonly used non-prescription medications and supplements were evaluated in ChatGPT using the query “Is X potentially harmful in people with kidney disease?” The resultant output was evaluated and categorized into one of three categories: “generally safe”, “potentially harmful”, or of “unknown" level of harm. Safety of the non-prescription medications and supplements was also evaluated in Micromedex and categorized similarly. Concordance between the two resources was summarized.

Results

Micromedex identified 68(55%) medications as safe, 52(42%) as potentially harmful, and 4(3%) as unknown. ChatGPT identified 74(60%) medications as safe, 26(21%) as potentially harmful, and 24(19%) as unknown. The overall agreement between Micromedex and ChatGPT was 65% with ChatGPT identifying only 46% of potentially harmful medications. Supplements as a subclass had the lowest concordance between ChatGPT and Micromedex, with a rate of 56%. Among the 24 medications identified as unknown by ChatGPT, 21(87%) were supplements.

Conclusion

ChatGPT’s ability to accurately assess the safety of non-prescription medications, particularly supplements, was modest in patients with kidney disease when compared to a contemporary drug information resource. The findings suggest that ChatGPT neither be considered nor recommended as a drug information resource for patients with kidney disease or their healthcare professionals. Further development would be necessary to improve its accuracy and reliability in this domain.