Kidney Week

Abstract: FR-PO0010

Development of a Large Language Model-Based Tool for Scoring Nephrology Clinical Reasoning

Session Information

• Artificial Intelligence: Clinical Prediction, Decision Support, and Digital Kidney Care
  October 23, 2026 | Location: Exhibit Hall A, Convention Center
  Abstract Time: 10:00 AM - 12:00 PM

Category: Artificial Intelligence, Digital Health, and Data Science

• 300 Artificial Intelligence, Digital Health, and Data Science

Authors

• Yang, Joanna, Icahn School of Medicine at Mount Sinai, New York, New York, United States

Joanna Yang, MS

• Martindale, James R., University of Virginia School of Medicine, Charlottesville, Virginia, United States

James R. Martindale, PhD

• Parsons, Andrew S., University of Virginia School of Medicine, Charlottesville, Virginia, United States

Andrew S. Parsons, MD, MPH

• Sozio, Stephen M., Johns Hopkins University Bloomberg School of Public Health, Baltimore, Maryland, United States

Stephen M. Sozio, MD, MEd, FASN

• Hilburg, Rachel, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania, United States

Rachel Hilburg, MD, FASN

• Stern, Lauren D., Boston University Chobanian & Avedisian School of Medicine, Boston, Massachusetts, United States

Lauren D. Stern, MD, MPH

• Warburton, Karen M., University of Virginia School of Medicine, Charlottesville, Virginia, United States

Karen M. Warburton, MD, FASN

• Boyle, Suzanne, Temple University Lewis Katz School of Medicine, Philadelphia, Pennsylvania, United States

Suzanne Boyle, MD, MS, FASN

• Chan, Lili, Icahn School of Medicine at Mount Sinai, New York, New York, United States

Lili Chan, MD, MS

Background

Clinical reasoning deficits contribute to diagnostic error and are common among nephrology fellows. We developed and tested a large language model (LLM) tool using a validated simulated acute kidney injury consult to assess reasoning in three domains: problem representation, differential diagnosis, and diagnostic plan.

Methods

We built a grading pipeline combining LLMs with rule-based calculations to assess fellows' written patient notes. GPT-OSS 20B used a dynamic, rubric-based prompt to grade responses. Seven human graders provided a majority consensus reference. Agreement between LLM and human scores was evaluated with Cohen's kappa and accuracy. Data were split into 80% (n=59) for prompt tuning and 20% (n=17) for testing. Two-way random effects ICC scored inter-rater reliability among human graders.

Results

Model performance varied across domains with high agreement in problem representation (test kappa range: 0.39-1.00; accuracy average 87.3%) and more moderate performance for differential diagnosis (test kappa range: 0.33-0.82, accuracy average 81.2%) and diagnostic plan (test kappa range: 0.04-0.64, accuracy average 73.5%). Lower ICC among raters was associated with poorer model-human agreement (Pearson's r=0.71, p<0.05) (Figure 1).

Conclusion

The model performed well on structured tasks (e.g. extracting and identifying clinical results) but was less reliable on complex, long-context reasoning (e.g., mapping justifications to diagnoses). It also lacked the flexibility of human graders, who considered all three domains and variations in clinical expressions. While these findings show promise for automated grading using an LLM, performance variability and lower agreement on complex tasks highlight the need for refinement in rubric language and human oversight.