Abstract: FR-PO0031
Assessment of OpenEvidence vs. ChatGPT-5 for Prior Authorization Letter Generation in Nephrology
Session Information
- Artificial Intelligence: Clinical Prediction, Decision Support, and Digital Kidney Care
October 23, 2026 | Location: Exhibit Hall A, Convention Center
Abstract Time: 10:00 AM - 12:00 PM
Category: Artificial Intelligence, Digital Health, and Data Science
- 300 Artificial Intelligence, Digital Health, and Data Science
Authors
- Aiumtrakul, Noppawit, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Thongprayoon, Charat, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Kookanok, Chutawat, One Brooklyn Health, New York, New York, United States
- Miao, Jing, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Cheungpasitporn, Wisit, Mayo Clinic Minnesota, Rochester, Minnesota, United States
Background
Writing prior authorization (PA) letters is a time-consuming task that can delay treatment initiation and increase clinician burden. Large language models (LLMs) may help facilitate PA workflows, but the quality of AI-generated letters should be evaluated. We assessed and compared the performance of ChatGPT-5 and OpenEvidence in generating PA letters for nephrology medications. Quality was evaluated based on factual accuracy and strength of clinical reasoning.
Methods
Twenty-eight nephrology clinical scenarios and a standardized prompt were entered into OpenEvidence and ChatGPT-5 to generate PA letters. We assessed each letter across 4 domains: presence of hallucinated or false statements, accuracy of ICD-10 coding, validity of cited references, and strength of clinical reasoning. FDA drug labels, KDIGO guidelines, and relevant trials served as reference standards. Comparisons were performed using the exact McNemar test. A descriptive review of error types was also performed.
Results
Across 28 paired scenarios, false statements occurred in 3.6% of ChatGPT-5 letters compared with 7.1% of OpenEvidence letters (p = 1.00). ICD-10 coding was correct in 78.6% of ChatGPT-5 letters versus 67.9% of OpenEvidence letters (p = 0.25). Reference accuracy was high for both models (92.9% vs 92.9%, p = 1.00). Clinical reasoning was rated strong in 89.3% of ChatGPT-5 letters and 78.6% of OpenEvidence letters (p = 0.45). Although none of the paired comparisons reached statistical significance, qualitative review revealed distinct error patterns. OpenEvidence consistently produced much longer texts, often exceeding the assigned 350-word range, and included several ICD 10 codes and references. Although this enhanced the clinical insight, the greater volume were also associated with a higher likelihood of inaccuracies, whereas ChatGPT 5 generated more concise drafts with fewer high impact mistakes. Importantly, hallucinations and factual inaccuracies were still observed in both LLMs.
Conclusion
LLMs could generate clinically usable PA letters, although error rates were similar across models, with differing error profiles. These findings support the use of LLM-generated letters as clinician-reviewed drafts prior to clinical use.