PuSH - Publication Server of Helmholtz Zentrum München: Performance of large language models as a source of clinical information on bacteriophage therapy.

Navigation

Home

Deutsch

Research

Advanced Search

Browse by ...

... Journal

... Publication Type

... Research Data

... Publication Year

Publication overview

Support & Contact

Contact persons

Help

Data protection

Walter, N. ; Amanatullah, D.F.* ; Debarbieux, L.* ; Doub, J.B.* ; Ferry, T.* ; Groß, J.* ; Międzybrodzki, R.* ; Mirzaei, M.K. ; Deng, L. ; Rācenis, K.* ; Suh, G.A.* ; Que, Y.* ; Górski, A.* ; Rupp, M.

Performance of large language models as a source of clinical information on bacteriophage therapy.

Npj Viruses 4:41 (2026)

Publ. Version/Full Text

Research data

DOI

PMC

	Open Access Gold

Abstract
Metrics
Extra information

Bacteriophage therapy is re-emerging as a potential strategy to address antimicrobial resistance, but standardized patient education materials are limited. Large language models (LLMs) are increasingly used for patient-facing medical information. The quality of LLM-generated responses to 20 patient-relevant questions was evaluated by 12 clinicians and research experts in bacteriophage therapy independently rated each response for accuracy, completeness, clarity, and tone/empathy using 5-point Likert scales. Expert suggestions for improvement were recorded. A total of 960 ratings were analyzed. Adjusted mean scores ranged from 3.36 to 3.96 across domains, indicating generally favorable evaluations for all models. Significant differences among LLMs were observed for completeness and tone/empathy (Holm-adjusted p = 0.042 for both), but not for accuracy or clarity. Differences were small in magnitude (Cohen's d = 0.12-0.29). Claude scored significantly lower than the other models for completeness and tone/empathy, while Perplexity achieved the highest completeness scores. Experts recommended improvements for 34-40% of responses; wrong information was given in 20%. The best responses were revised into an expert-informed patient guide provided as Supplementary Material, presenting a hybrid model in which LLMs generate draft patient information that is subsequently refined by clinical experts, particularly in rapidly evolving therapeutic domains lacking standardized educational resources.

Altmetric

Additional Metrics?

[➜Log in]

Edit extra informations Login

Publication type Article: Journal article

Document type Scientific Article

Keywords Perplexity ; Completeness (order Theory) ; Likert Scale ; Quality (philosophy) ; Language Model ; Medline ; Health Care

ISSN (print) / ISBN 2948-1767

e-ISSN 2948-1767

Journal Npj Viruses

Quellenangaben Volume: 4, Issue: 1, Article Number: 41

Publisher Springer

Reviewing status Peer reviewed

Institute(s) Institute of Virology (VIRO)