Summaries generated by large language models enhance communication across different disciplines in ophthalmology

(Image credit: ©Pakin/AdobeStock)

The results of a large randomized, quality improvement study in the US suggested that integrating large language model (LLM)-generated plain language summaries into ophthalmology notes can improve interdisciplinary communication, although ongoing review and safety monitoring are necessary for successful clinical implementation,¹ according to the first author, Prashant D. Tailor, MD. He is associated with the Department of Ophthalmology, Mayo Clinic, Rochester, MN, and the Jules Stein Eye Institute and Department of Ophthalmology, David Geffen School of Medicine at UCLA, University of California, Los Angeles, Los Angeles.

In addition to other coauthors from the Department of Ophthalmology, Mayo Clinic, Rochester, he was joined in this study by investigators from the Department of Allergy & Immunology, ICAHN School of Medicine at Mount Sinai, New York; the Department of Quantitative Health Sciences, Mayo Clinic, Jacksonville, FL; and the Department of Neurology, Mayo Clinic, Rochester.

The authors explained that “the specialized nature and unique terminology of ophthalmology create a comprehension gap for non-ophthalmology clinicians and professionals such as internists and primary care physicians. This gap can hinder interdisciplinary communication and impact patient care. Current solutions, such as manual note simplification or abbreviation tables, are impractical and insufficient. LLMs offer a potential solution to this challenge,” they said.

Quality improvement study

The study included data from inpatient and outpatient encounters in one tertiary academic center. The participants were non-ophthalmology clinicians and professionals and ophthalmologists. The inclusion criterion was any encounter note generated by an ophthalmologist during the study dates, which ranged from February 1, 2024, to May 31, 2024. The exclusion criteria were the lack of established non-ophthalmology clinicians and professionals for outpatient encounters and procedure-only patient encounters.

LLM-generated plain language summaries were added to the ophthalmology notes. The main outcome was the survey responses from non-ophthalmology clinicians and professionals; the survey evaluated the understanding, satisfaction, and clarity of the ophthalmology notes. The secondary outcomes were survey responses from ophthalmologists who evaluated the plain language summaries regarding clinical workflow and accuracy, objective measures of semantic quality, and safety analysis.

Specific findings

The results showed that 362 (85%) non-ophthalmology clinicians and professionals (33.0% response rate) preferred the plain language summaries to the standard ophthalmology notes. Non-ophthalmology clinicians and professionals reported enhanced diagnostic understanding (percentage point increase, 9.0; 95% confidence interval [CI], 0.3-18.2; P = 0.01), increased note detail satisfaction (percentage point increase, 21.5; 95% CI, 11.4-31.5; P < 0.001), and improved explanation clarity (percentage point increase, 23.0; 95% CI, 12.0-33.1; P < 0.001) for notes containing a plain language summary, the authors reported in JAMA Ophthalmology.

“The addition of a plain language summary was associated with reduced comprehension gaps between clinicians who were comfortable and uncomfortable with ophthalmology terminology from 26.1% (95% CI, 13.7%-38.6%; P < 0.001) to 14.4% (95% CI, 4.3%-24.6%; P > 0.06). Analysis of the plain language summary semantics found high meaning preservation with greater readability than standard ophthalmology notes.

The ophthalmologists (n = 489; 84% response rate) who participated reported high accuracy with the plain language summaries (90% [320 of 355] reported a great deal) with minimal review time burden (94.9% [464 of 489] ≤1 minute). The plain language summary error rate on ophthalmologist review was 26% (126 of 489). A total of 83.9% (104 of 126) of errors were deemed low risk for harm and none had a risk of severe harm or death.

“Plain language summaries have the potential to enhance interdisciplinary communication and make medical information more accessible, improving collaborative patient care. Future research should test LLM-generated summaries in other specialties and settings to validate this approach. The presence of errors in AI-generated summaries despite human oversight highlights the need for careful ongoing safety monitoring to mitigate any patient safety concerns prior to clinical implementation,” Dr. Tailor and colleagues concluded.

Reference

1. Tailor PD, D’Souza HS, Becerra CMC, et al. Evaluation of AI Summaries on interdisciplinary understanding of ophthalmology notes. JAMA Ophthalmol. 2025; Published online April 3, 2025. doi:10.1001/jamaophthalmol.2025.0351