Kidney Week

Abstract: SA-PO0064

Rich Clinical Context, Not Just Alignment, Drives Multimodal MRI Model Utility

Session Information

• Artificial Intelligence: Kidney Imaging, Pathology, and Precision Phenotyping
  October 24, 2026 | Location: Exhibit Hall A, Convention Center
  Abstract Time: 10:00 AM - 12:00 PM

Category: Artificial Intelligence, Digital Health, and Data Science

• 300 Artificial Intelligence, Digital Health, and Data Science

Authors

• Dhar, Mrinal Kanti, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Mrinal Kanti Dhar, PhD

• Gregory, Adriana, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Adriana Gregory, MS

• Buchl, Samuel, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Samuel Buchl

• Green, Abigail, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Abigail Green

• Holmes, Heather L., Mayo Clinic Minnesota, Rochester, Minnesota, United States

Heather L. Holmes

• Fati, Francesca, Mayo Clinic Minnesota, Rochester, Minnesota, United States

Francesca Fati, MS

• Kline, Timothy L., Mayo Clinic Minnesota, Rochester, Minnesota, United States

Timothy L. Kline, MS, PhD

Background

Optimal image-text pairing strategies for multimodal pretraining in abdominal MRI remain unclear, particularly whether stronger alignment improves downstream classification performance or clinical utility.

Methods

Three multimodal foundation models were pretrained by contrastive learning on 28,187 abdominal MRI-text pairs using identical architectures that differed only by text source: full radiology reports, structured clinical annotations, or technical metadata/acquisition descriptions. Images were encoded with a 3D Vision Transformer and text with a radiology-pretrained CXR-BERT. Frozen image encoders were linearly probed for liver cyst, kidney cyst, and chronic kidney disease (CKD) classification. CKD was defined using the nearest serum eGFR within 15 days of MRI (<60 vs >=60 mL/min/1.73 m²). To assess representation alignment, cosine similarity was measured between image embeddings and matched text, random unmatched text, and same-label unmatched text (random text sharing the same label).

Results

Free-text pretraining yielded the best downstream performance. For kidney cyst detection, sensitivity/AUC were 0.97/0.67 vs 0.78/0.66 (structured) and 0.50/0.62 (metadata); for liver cyst, 0.94/0.73 vs 0.72/0.70 and 0.55/0.64. For CKD, free-text improved accuracy/sensitivity (0.67/0.86) vs 0.59/0.81 and 0.64/0.85, with AUC 0.70 (vs 0.72 for metadata). In contrast, metadata showed the strongest alignment (cosine similarity 0.93 vs 0.53 and 0.20), while unmatched pairs remained near zero, indicating that stronger alignment did not consistently translate to better downstream performance.

Conclusion

For multimodal radiology AI, maximizing image-text similarity is insufficient; contextual richness of full reports may be more important for building clinically useful foundation models.

Funding

• Private Foundation Support