Abstract: FR-PO0039
Large Language Model-Assisted Screening in a Systematic Review on the Use, Administration, and Interpretation of Patient-Reported Outcome Measures in Dialysis Randomized Controlled Trials
Session Information
- Artificial Intelligence: Clinical Prediction, Decision Support, and Digital Kidney Care
October 23, 2026 | Location: Exhibit Hall A, Convention Center
Abstract Time: 10:00 AM - 12:00 PM
Category: Artificial Intelligence, Digital Health, and Data Science
- 300 Artificial Intelligence, Digital Health, and Data Science
Authors
- Zdunek, Victoria, Western University, London, Ontario, Canada
- Klotz, Adam, Western University, London, Ontario, Canada
- Bu, Shuhan, Western University, London, Ontario, Canada
- Rueda, Maria D., Western University, London, Ontario, Canada
- Leung, Caitlin, Western University, London, Ontario, Canada
- Roshanov, Pavel, Western University, London, Ontario, Canada
Background
Patients receiving dialysis prioritize patient-reported outcome measures (PROMs), yet trials that report these outcomes are difficult to identify. PROM reporting in abstracts is inconsistent and often indirect (e.g., referencing symptoms or quality of life without naming an instrument), reducing keyword-based screening and increasing reviewer workload. Large Language Models (LLMs) may improve screening efficiency by applying structured eligibility criteria, but their ability to detect PROM-related content requires evaluation. We evaluated whether LLMs accurately identify dialysis trials reporting PROMs.
Methods
We generated a reference standard consisting of 172 eligible records from a completed systematic review and 516 ineligible records randomly sampled from the same screening pool. Eleven open-weight models and one closed-weight frontier model were evaluated. Models independently screened titles and abstracts using predefined criteria. A one-at-a-time approach was used, where each model screened one criterion at a time to determine overall eligibility. Model performance metrics and potential workload reduction were calculated against the human gold standard.
Results
Eligibility prevalence was 25%. Overall, the strongest performing model was open-weight GPT-OSS-120b Q8, achieving sensitivity of 92.5% (95% CI: 87.6-95.6), specificity 94.0% (91.6-95.7), PPV 83.8% (77.9-88.3), and NPV 97.4% (95.6-98.5). The closed-weight Gemini 3 Flash showed similar overall performance (accuracy 92.4%, 90.2-94.2). GPT-OSS-120b Q8 reached consensus on 93.6% (644/688) of records; the 44 discordant records (6.4%) were directed to human review. Automatically excluding model-classified ineligible records reduced human screening workload by 72.2%.
Conclusion
LLM-assisted screening can identify PROM-reporting trials with high accuracy and significantly reduce reviewer workload. Open-weight models performed comparably to closed-weight frontier models and achieved the highest overall accuracy. However, these models should be used as a decision-support tool rather than a standalone reviewer.