PuSH - Publication Server of Helmholtz Zentrum München

Schaper, B.* ; di Folco, M. ; Kainz, B.* ; Schnabel, J.A. ; Bercea, C.-I.

Measuring and aligning abstraction in vision-language models with medical taxonomies.

In: (23rd IEEE International Symposium on Biomedical Imaging, ISBI 2026, 8-11 April 2026, London). 2026. (Proceedings International Symposium on Biomedical Imaging ; 2026-April)
DOI
Vision-Language Models (VLMs) show strong zero-shot performance for chest X-ray classification, but standard flat metrics fail to distinguish between clinically minor and severe errors. This work investigates how to quantify and mitigate abstraction errors by leveraging medical taxonomies. We benchmark several state-of-the-art VLMs using hierarchical metrics and introduce Catastrophic Abstraction Errors to capture cross-branch mistakes. Our results reveal substantial misalignment of VLMs with clinical taxonomies despite high flat performance. To address this, we propose risk-constrained thresholding and taxonomy-aware fine-tuning with radial embeddings, which reduce severe abstraction errors to below 2% while maintaining competitive performance. These findings highlight the importance of hierarchical evaluation and representation-level alignment for safer and more clinically meaningful deployment of VLMs.
Altmetric
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Conference contribution
Keywords Cxr ; Hierarchical Metrics ; Vision-language Models ; Zero-shot Classification
ISSN (print) / ISBN 1945-7928
e-ISSN 1945-8452
Conference Title 23rd IEEE International Symposium on Biomedical Imaging, ISBI 2026
Conference Date 8-11 April 2026
Conference Location London
Quellenangaben Volume: 2026-April Issue: , Pages: , Article Number: , Supplement: ,
Institute(s) Institute for Machine Learning in Biomed Imaging (IML)