Abstract: FR-PO0030
Evaluating Large Language Models for Detection of Renoprotective Therapy Omissions in CKD: A Synthetic Benchmark Study
Session Information
- Artificial Intelligence: Clinical Prediction, Decision Support, and Digital Kidney Care
October 23, 2026 | Location: Exhibit Hall A, Convention Center
Abstract Time: 10:00 AM - 12:00 PM
Category: Artificial Intelligence, Digital Health, and Data Science
- 300 Artificial Intelligence, Digital Health, and Data Science
Author
- Lin, Hsuan Ming, Tainan Municipal An Nan Hospital, Tainan City, Taiwan
Background
SGLT2 inhibitors, finerenone, and RASi remain underutilized in CKD (real-world SGLT2i use ~14.8%). Large language models (LLMs) may help identify therapeutic omissions, but performance across CKD stages and at clinical decision boundaries is unknown.
Methods
We constructed 100 case vignettes spanning CKD stages 3a–5D (n=30/33/37) with predefined renoprotective omissions per KDIGO 2024 eligibility criteria: SGLT2 inhibitor (n=36), finerenone (n=34), RASi (n=35; including prespecified eGFR<15 boundary cases), and 25 cases without true treatment gaps. Four LLMs (GPT-5.4, Gemini 3.1 Pro, Grok 4.1 Fast, DeepSeek-R1) reviewed each vignette under a structured clinical query at deterministic settings; Claude Sonnet 4.6 served as case generator and was excluded from evaluation. Three queries per case yielded reproducible outputs; analyses used one decision per model-case. Primary outcomes: sensitivity per gap type and false-positive rate, with Wilson 95% CIs.
Results
Sensitivity was 100% for SGLT2 inhibitor and finerenone omissions across three models (94–97% for DeepSeek-R1). RASi detection varied substantially (88.6%, 74.3%, 45.7%, 45.7%; p<0.001 by Cochran's Q), driven by eGFR<15 boundary cases where models often missed RASi omission in prespecified candidates without contraindications (95%, 45%, 0%, 15%). Specificity in cases without true gaps remained high (88–100%).
Conclusion
Frontier LLMs reliably detect SGLT2 inhibitor and finerenone omissions with a structured clinical query, but show pronounced conservative bias at the eGFR<15 RASi decision boundary—a clinical gray zone with limited explicit guideline guidance. Clinicians evaluating LLM-based decision support for CKD should benchmark performance at advanced-stage RASi decisions, where conservative bias may lead to under-recommendation of guideline-supported therapy.
Acknowledgment
Synthetic-data study per KDIGO 2024; IRB-exempt. Four LLMs (GPT-5.4, Gemini 3.1 Pro, Grok 4.1 Fast, DeepSeek-R1) were evaluated via OpenRouter. Claude Sonnet 4.6 served as case generator and was excluded from evaluation. All outputs were classified against a predefined gold standard using a structured rubric and verified by the principal investigator, who bears full responsibility for clinical content. Manuscript preparation assisted by Claude Code. No external funding; no conflicts of interest.