PuSH - Publication Server of Helmholtz Zentrum München: Illusion of competence: Vision–language models provide confident but inaccurate explanations in cytological diagnostics.

Navigation

Home

Deutsch

Research

Advanced Search

Browse by ...

... Journal

... Publication Type

... Research Data

... Publication Year

Publication overview

Support & Contact

Contact persons

Help

Data protection

Kukuljan, I. ; Dasdelen, M.F. ; Schäfer, J. ; Buck, M.* ; Götze, K.* ; Marr, C.

Illusion of competence: Vision–language models provide confident but inaccurate explanations in cytological diagnostics.

Sci. Rep. 16:20526 (2026)

Publ. Version/Full Text

DOI

PMC

	Open Access Gold

Abstract
Metrics
Extra information

Large vision-language models (LVLMs) have shown impressive image-understanding capabilities across domains. However, their suitability for cytomorphological diagnostics remains unclear. Here, we systematically evaluated four state-of-the-art generalist LVLMs, GPT-4o, Gemini-2.0, Llama-3.2, and DeepSeek-VL2, and three biomedical LVLMs, LLaVA-Med, CONCH, and BiomedCLIP, across key cytomorphology benchmarks, including peripheral blood cell classification, morphology assessment, bone marrow cell classification, and cervical smear malignancy detection. Performance was assessed under zero-shot, few-shot, and fine-tuned settings. In zero-shot and few-shot evaluations, LVLMs performed poorly, often approaching random performance. In peripheral blood cell classification, GPT-4o achieved a zero-shot F1 score of only 0.22 ± 0.02 and a few-shot F1 score of 0.36 ± 0.03. Even after fine-tuning, GPT-4o was outperformed by a lightweight, dedicated hematology model. Beyond classification accuracy, we assessed interpretability and trustworthiness. Although LVLMs generated textual justifications, these often reflected textbook knowledge rather than the actual morphological features present in the cell images. Expert evaluation showed that 30% of explanations for misclassified cells were rated as poor or misleading. While LVLMs could segment cellular structures such as nuclei and granules, they failed to reliably identify the image regions relevant to their classification decisions. Our findings underscore three major limitations of current LVLMs in cytomorphology: (1) low diagnostic accuracy, (2) poor generalizability across domains, and (3) unreliable explainability. These results suggest that LVLMs require substantial improvement before they can be used for cell-type classification and morphology characterization in diagnostic settings. Purpose-built models remain the more effective and trustworthy choice.

Altmetric

Additional Metrics?

[➜Log in]

Edit extra informations Login

Publication type Article: Journal article

Document type Scientific Article

Keywords Interpretability ; Generalizability Theory ; Blood Film ; Trustworthiness ; Malignancy ; Illusion

ISSN (print) / ISBN 2045-2322

e-ISSN 2045-2322

Journal Scientific Reports

Quellenangaben Volume: 16, Issue: 1, Article Number: 20526

Publisher Springer

Publishing Place London

Reviewing status Peer reviewed

Institute(s) Institute of AI for Health (AIH)