Abstract: FR-PO0024
Evaluation of Large Language Models for AKI Risk Factor Extraction from Hospital Discharge Notes
Session Information
- Artificial Intelligence: Clinical Prediction, Decision Support, and Digital Kidney Care
October 23, 2026 | Location: Exhibit Hall A, Convention Center
Abstract Time: 10:00 AM - 12:00 PM
Category: Artificial Intelligence, Digital Health, and Data Science
- 300 Artificial Intelligence, Digital Health, and Data Science
Authors
- Wico, Marcvincent C., University of California San Diego, La Jolla, California, United States
- Nguyen, Ashley, University of California San Diego, La Jolla, California, United States
- Nguyen, Jonathan D., University of California San Diego, La Jolla, California, United States
- Tran, Clayton, University of California San Diego, La Jolla, California, United States
- Yousif, Zaid, University of California San Diego, La Jolla, California, United States
- Awdishu, Linda, University of California San Diego, La Jolla, California, United States
Background
Extracting acute kidney injury (AKI) risk factors from narrative clinical notes is a time-intensive task. We hypothesize that large language models (LLMs) can efficiently extract AKI risk factors from narrative discharge clinical notes with performance comparable to that of supervised pre-clinical students.
Methods
We selected a random sample of 100 discharge summaries from adult patients hospitalized at the University of California San Diego Health system from 1/1/2013 to 12/31/2023. Inclusion criteria were age ≥18 years and development of AKI stage II or higher, as defined by the Kidney Disease: Improving Global Outcomes (KDIGO) criteria. Patients with a history of kidney or bone marrow transplantation or with end-stage kidney disease were excluded.
For each hospitalization, AKI risk factors were manually extracted from discharge notes by one of two independent teams, each consisting of two pharmacy students. Each team reviewed a distinct set of discharge notes, and the resulting annotations served as the ground truth. A zero-shot comprehensive prompt was used to extract AKI risk factors from narrative discharge notes using the Anthropic Claude 3.7 LLM. AKI risk factors with discordant classifications between manual extraction and the LLM extraction underwent secondary adjudication by a pharmacist to determine the final present/absent status of each AKI risk factor.
Results
A total of 57 AKI risk factors were evaluated. Compared with manual extraction, the LLM demonstrated high performance, achieving a sensitivity of 0.978, specificity of 0.988, positive predictive value of 0.905, and negative predictive value of 0.997. The F1 score was 0.940, and interrater reliability, measured by Cohen's kappa, was 0.936. Figure 1 illustrates the prevalence and LLM performance of AKI risk factors across different categories.
Conclusion
This analysis demonstrates the potential utility of LLMs for extracting AKI risk factors from narrative clinical notes. Future work will focus on evaluating LLM performance in a larger sample size.