Abstract: TH-PO007
The Accuracy of Artificial Intelligence in Identifying Potentially Harmful Non-Prescription Medications and Dietary Supplements in Patients with Kidney Diseases
Session Information
- AI, Digital Health, Data Science - I
November 02, 2023 | Location: Exhibit Hall, Pennsylvania Convention Center
Abstract Time: 10:00 AM - 12:00 PM
Category: Augmented Intelligence, Digital Health, and Data Science
- 300 Augmented Intelligence, Digital Health, and Data Science
Authors
- Sheikh, M. Salman, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Barreto, Erin F., Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Dreesman, Benjamin, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Qureshi, Fawad, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Gregoire, James Robert, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Erickson, Stephen B., Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Thongprayoon, Charat, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Craici, Iasmina, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Cheungpasitporn, Wisit, Mayo Clinic Minnesota, Rochester, Minnesota, United States
Background
Search engines are commonly used to obtain health-related information, including information about drug safety. Non-prescription medications and supplements are generally perceived as safe, however, may be harmful to patients with kidney diseases. ChatGPT is a cutting-edge language model that has gained attention for its potential to improve clinical decision making. Its ability to serve as a drug information resource for patients with kidney disease has not been determined. This study aimed to evaluate ChatGPT’s accuracy in discerning the safety of medications in patients with kidney diseases when compared to Micromedex, a widely used tertiary drug information reference.
Methods
One hundred twenty-four commonly used non-prescription medications and supplements were evaluated in ChatGPT using the query “Is X potentially harmful in people with kidney disease?” The resultant output was evaluated and categorized into one of three categories: “generally safe”, “potentially harmful”, or of “unknown" level of harm. Safety of the non-prescription medications and supplements was also evaluated in Micromedex and categorized similarly. Concordance between the two resources was summarized.
Results
Micromedex identified 68(55%) medications as safe, 52(42%) as potentially harmful, and 4(3%) as unknown. ChatGPT identified 74(60%) medications as safe, 26(21%) as potentially harmful, and 24(19%) as unknown. The overall agreement between Micromedex and ChatGPT was 65% with ChatGPT identifying only 46% of potentially harmful medications. Supplements as a subclass had the lowest concordance between ChatGPT and Micromedex, with a rate of 56%. Among the 24 medications identified as unknown by ChatGPT, 21(87%) were supplements.
Conclusion
ChatGPT’s ability to accurately assess the safety of non-prescription medications, particularly supplements, was modest in patients with kidney disease when compared to a contemporary drug information resource. The findings suggest that ChatGPT neither be considered nor recommended as a drug information resource for patients with kidney disease or their healthcare professionals. Further development would be necessary to improve its accuracy and reliability in this domain.