Abstract: FR-PO0026
Evaluating the Accuracy and Reasoning of Large Language Models (LLM) in Selecting Statistical Methodologies for Nephrology Clinical Trials
Session Information
- Artificial Intelligence: Clinical Prediction, Decision Support, and Digital Kidney Care
October 23, 2026 | Location: Exhibit Hall A, Convention Center
Abstract Time: 10:00 AM - 12:00 PM
Category: Artificial Intelligence, Digital Health, and Data Science
- 300 Artificial Intelligence, Digital Health, and Data Science
Authors
- Fichadiya, Harshil, Mayo Clinic Minnesota, Rochester, Minnesota, United States
- Cheungpasitporn, Wisit, Mayo Clinic Minnesota, Rochester, Minnesota, United States
Background
Appropriate statistical methodology is critical to clinical trial validity, yet its selection remains a well-recognized challenge among clinician-investigators. LLMs are increasingly utilized in clinical research; however, their ability to correctly identify nuanced statistical methodologies across diverse trial designs remains underexplored. Objective: To evaluate and compare the accuracy of ChatGPT, Microsoft Copilot, and Claude AI in selecting the correct primary statistical method across landmark nephrology clinical trials.
Methods
A cross-sectional comparative assessment was performed using 6 blinded prompts derived from landmark nephrology trials (EMPA-Kidney, ORIGIN-3, VALOR-CKD, SONAR, CKD-PC, and the CRIC Heart Study). Trial-identifying information was systematically removed to prevent database retrieval, ensuring responses reflected intrinsic statistical reasoning rather than pattern recognition from known publications. Each prompt outlined trial design, objectives, and primary endpoints, with models choosing from: Cox Proportional Hazards, Linear Mixed-Effects, Random-Effects Meta-Analysis, Logistic Regression, Fine-Gray, and ANCOVA options. Accuracy was benchmarked against the method reported in each trial's primary publication, and qualitative assessment of model justifications was performed to evaluate alignment with core statistical principles.
Results
All three LLMs achieved 100% accuracy across all 6 trials, correctly identifying appropriate methods for time-to-event composites (Cox), fixed-timepoint continuous endpoints (ANCOVA), longitudinal kidney function trajectories (linear mixed-effects), competing risk analyses (Fine-Gray), and cross-cohort pooling (random-effects meta-analysis). Models consistently provided structured justifications grounded in key statistical concepts.
Conclusion
LLMs demonstrated high concordance with published statistical methodologies across selected landmark nephrology trials under constrained conditions.
Results
| Study | Primary Outcome | Method used by authors | ChatGPT | Microsoft Co-Pilot | Claude AI | Reasoning provided by AI |
| EMPA-Kidney | Time to ESKD/CV Death | Cox Proportional Hazards | Cox Proportional Hazards | Cox Proportional Hazards | Cox Proportional Hazards | Identified "time-to-first-event" composite logic |
| ORIGIN-3 | % Change in UPCR (Wk 36) | ANCOVA | ANCOVA | ANCOVA | ANCOVA | Recognized fixed-point continuous variable adjusted for baseline. |
| VALOR-CKD | Annualized eGFR Slope | Linear Mixed-Effects | Linear Mixed-Effects | Linear Mixed-Effects | Linear Mixed-Effects | Accounted for repeated measures and nested data. |
| SONAR | Kidney Failure vs. Death | Fine-Gray Model | Fine-Gray Model | Fine-Gray Model | Fine-Gray Model | Identified death as a competing risk for renal events. |
| CKD-PC | Pooled Hazard Ratios | Random-Effects Meta-Analysis | Random-Effects Meta-Analysis | Random-Effects Meta-Analysis | Random-Effects Meta-Analysis | Recognized the synthesis of independent cohort data. |
| CRIC Heart | Incident KFRT (Cohort) | Multivariable Cox | Multivariable Cox | Multivariable Cox | Multivariable Cox | Linked longitudinal observational data to time-to-event risk. |