Abstract: FR-PO0029
Evaluating Large Language Models for CKD Medication Review: A Trap-Embedded Benchmark
Session Information
- Artificial Intelligence: Clinical Prediction, Decision Support, and Digital Kidney Care
October 23, 2026 | Location: Exhibit Hall A, Convention Center
Abstract Time: 10:00 AM - 12:00 PM
Category: Artificial Intelligence, Digital Health, and Data Science
- 300 Artificial Intelligence, Digital Health, and Data Science
Author
- Lin, Hsuan Ming, Tainan Municipal An Nan Hospital, Tainan City, Taiwan
Background
Chronic kidney disease (CKD) patients with polypharmacy require comprehensive medication review addressing renal dosing, drug interactions, deprescribing, and stage-dependent contraindications. Large language models (LLMs) show promise for clinical decision support, but nephrology-specific evaluation methods remain limited. We developed a trap-embedded benchmark for LLM medication review across CKD severity (Figure 1).
Methods
Fifty synthetic CKD cases were constructed from prespecified templates across three tiers: G3a-G3b (n=20), G4 (n=15), and G5/G5D/transplant (n=15), with 8-12 medications each. Ten clinical trap categories were embedded. Gold standards were developed per Kidney Disease: Improving Global Outcomes 2024 and STOPP/START v3, adjudicated by a board-certified nephrologist. Four LLMs (GPT-5.4, Claude Sonnet 4.6, Gemini 3.1 Pro, Grok 4.1 Fast) reviewed all cases in triplicate at temperature 0 (600 outputs). Scoring used LLM-assisted extraction followed by nephrologist validation. Outcomes: recall, precision, F1, safety-critical error rate (SCE; potentially harmful errors), hallucination rate, and consistency (Jaccard index).
Results
F1 ranged from 0.63 to 0.72 (Friedman p=0.003; Figure 2). Three critical traps — NSAID contraindication, metformin in advanced CKD, and tacrolimus-azole interaction — were detected in 100% of cases; other traps 77-97%. Phosphate binder timing errors were detected in only 14%, statin indication gaps in 65%. SCE rates: 16-42% (Cochran Q p=0.006). Hallucination rates: 0-76%. Consistency: Jaccard 0.40-0.77, indicating instability despite deterministic settings.
Conclusion
This benchmark provides a reproducible, nephrology-specific method for evaluating LLM medication review, identifying blind spots — particularly administration timing and preventive therapy gaps — missed by generic benchmarks. Wide safety and consistency variation supports specialty-specific validation before clinical deployment.
Acknowledgment
This study received no external funding. API costs were self-funded. Generative AI tools were used as both research subjects (four LLMs evaluated) and research tools (Claude Sonnet 4.5 for scoring extraction; Claude Code for analysis pipeline). All AI outputs were validated by the principal investigator.