Abstract: FR-PO0038
A Large Language Model-Driven Strategy for High-Precision Abstract Screening in Systematic Reviews
Session Information
- Artificial Intelligence: Clinical Prediction, Decision Support, and Digital Kidney Care
October 23, 2026 | Location: Exhibit Hall A, Convention Center
Abstract Time: 10:00 AM - 12:00 PM
Category: Artificial Intelligence, Digital Health, and Data Science
- 300 Artificial Intelligence, Digital Health, and Data Science
Authors
- Bergling, Karin, Renal Research Institute, New York, New York, United States
- Yueh, Sheng-Han, Renal Research Institute, New York, New York, United States
- Lama, Suman Kumar, Renal Research Institute, New York, New York, United States
- Willetts, Joanna, Renal Research Institute, New York, New York, United States
- Blankenship, Derek M., Renal Research Institute, New York, New York, United States
- Winter, Anke, Renal Research Institute, New York, New York, United States
- Usvyat, Len A., Renal Research Institute, New York, New York, United States
- Raimann, Jochen G., Fresenius Medical Care Holdings Inc, Waltham, Massachusetts, United States
- Zhang, Hanjie, Renal Research Institute, New York, New York, United States
Background
Abstract screening is resource-intensive in systematic reviews and a key bottleneck in evidence synthesis informing clinical care. We evaluated whether GPT-5.1 (OpenAI) could support a renal-focused abstract screening workflow while adhering to predefined Population–Intervention–Comparator–Outcome–Study (PICOS) eligibility criteria.
Methods
We analyzed 1,628 abstracts from a prior systematic review examining adverse effects of systemic heparin in maintenance hemodialysis. Abstracts labeled “keep” or “discard” according to PICOS criteria were divided into validation (n=277, 17%) and test (n=1,351, 83%) sets. PICOS criteria were operationalized into seven screening questions. GPT-5.1 responded “yes,” “no,” or “uncertain,” and responses were algorithmically combined to generate final “keep” or “discard” classifications. To prioritize sensitivity, all “uncertain” outputs were retained as “keep,” ensuring only clearly irrelevant studies were excluded. Model performance was evaluated against independent dual human screens.
Results
In validation, overall accuracy was 90%. For “discard,” precision was 0.99 and recall 0.90. “Keep” decisions showed lower precision (0.51) but high recall (0.93), reflecting conservative retention of uncertain cases. In the test set, GPT-5.1 flagged 35 abstracts for exclusion initially retained; human reassessment confirmed 20 as irrelevant, prompting revision of the reference labeling. After updating, accuracy was 87%. For “discard,” precision and recall were 0.99 and 0.87 (F1=0.92); for “keep,” 0.42 and 0.89. The confusion matrix comprised 1,058 true negatives, 116 true positives, 162 false positives, and 15 false negatives. Manual workload decreased by 79%, with 278 of 1,351 abstracts requiring review.
Conclusion
GPT-5.1 effectively identified abstracts appropriate for exclusion while a recall-oriented design minimized missed relevant studies, enabling human reviewers to focus on potentially eligible evidence.
Acknowledgment
Large language models were used as part of the formal abstract screening method and to assist with abstract preparation. AI-generated outputs were reviewed and verified by the authors, who take full responsibility for the integrity and accuracy of the submitted content.
Figure 1. A. Agreement between GPT-5.1 and human reviewers in the test set (n=1,351). B. Manual review, automated exclusions, and missed relevant cases.
Funding
- Commercial Support – Fresenius Medical Care