Kidney Week

Abstract: FR-PO0036

Comparative Performance of Five Large Language Models in Systematic Review Abstract Screening

Session Information

• Artificial Intelligence: Clinical Prediction, Decision Support, and Digital Kidney Care
  October 23, 2026 | Location: Exhibit Hall A, Convention Center
  Abstract Time: 10:00 AM - 12:00 PM

Category: Artificial Intelligence, Digital Health, and Data Science

• 300 Artificial Intelligence, Digital Health, and Data Science

Authors

• Bergling, Karin, Renal Research Institute, New York, New York, United States

Karin Bergling, MD, PhD

• Winter, Anke, Renal Research Institute, New York, New York, United States

Anke Winter, MD

• Yueh, Sheng-Han, Renal Research Institute, New York, New York, United States

Sheng-Han Yueh, MS

• Lama, Suman Kumar, Renal Research Institute, New York, New York, United States

Suman Kumar Lama, MS, MSc

• Willetts, Joanna, Renal Research Institute, New York, New York, United States

Joanna Willetts

• Blankenship, Derek M., Renal Research Institute, New York, New York, United States

Derek M. Blankenship, PhD, MBA

• Raimann, Jochen G., Renal Research Institute, New York, New York, United States

Jochen G. Raimann, MD, PhD, MPH, FASN

• Usvyat, Len A., Renal Research Institute, New York, New York, United States

Len A. Usvyat, PhD, FASN

• Zhang, Hanjie, Renal Research Institute, New York, New York, United States

Hanjie Zhang, PhD

Background

Abstract screening is a major workload in systematic reviews. Large language models (LLMs) may reduce this burden by identifying non-relevant studies while preserving eligible records. We hypothesized that newer LLMs would outperform earlier models.

Methods

We evaluated 1,351 abstracts from a systematic review of heparin-induced side effects in maintenance hemodialysis. Five OpenAI LLMs, gpt-4o, gpt-4.1, o1, gpt-5.4-mini, and gpt-5.4, were given brief study context and predefined Population, Intervention, Comparator, Outcome, and Study design (PICOS) criteria. Each abstract was classified as “keep,” “discard,” or “uncertain”; “uncertain” was treated as “keep.” Classifications were compared with a human reference standard from two independent reviewers. Human-kept abstracts discarded by at least one LLM were re-reviewed to distinguish true LLM false negatives from overly inclusive human keeps.

Results

Among 126 re-reviewed abstracts, 59 were reclassified as exclude, yielding a final reference standard of 92 included and 1,259 excluded. The performance of the five LLMs is summarized in Table 1. o1 was least likely to miss relevant studies, identifying 68 of 92 abstracts and missing 24. gpt-5.4 was most selective, incorrectly retaining only 21 of 1,259 excluded abstracts, but missed 59 relevant abstracts. The earlier gpt-4o model showed a similar sensitivity profile to gpt-5.4, also missing 59 relevant abstracts, but retained slightly more excluded abstracts. Fig. 1 illustrates this trade-off across models.

Conclusion

Performance varied by model, and newer models were not consistently better. All missed relevant studies, falling short of independent screening. LLMs may support screening workflows, but human oversight remains essential.

Acknowledgment

Large language models were used as part of the abstract screening workflow described in this study and to assist with abstract preparation. The authors take full responsibility for the accuracy and final content of the abstract.

LLM screening performance.

Missed relevant vs retained non-relevant abstracts.

Funding

• Commercial Support – Fresenius Medical Care