PuSH - Publication Server of Helmholtz Zentrum München

Müller, P.* ; Kaissis, G. ; Rueckert, D.*

ChEX: Interactive Localization and Region Description in Chest X-Rays.

In: (18th European Conference on Computer Vision, ECCV 2024, 29 September - 4 October 2024, Milan). Berlin [u.a.]: Springer, 2025. 92-111 (Lect. Notes Comput. Sc. ; 15079 LNCS)
DOI
Open Access Green as soon as Postprint is submitted to ZB.
Report generation models offer fine-grained textual interpretations of medical images like chest X-rays, yet they often lack interactivity (i.e. the ability to steer the generation process through user queries) and localized interpretability (i.e. visually grounding their predictions), which we deem essential for future adoption in clinical practice. While there have been efforts to tackle these issues, they are either limited in their interactivity by not supporting textual queries or fail to also offer localized interpretability. Therefore, we propose a novel multitask architecture and training paradigm integrating textual prompts and bounding boxes for diverse aspects like anatomical regions and pathologies. We call this approach the Chest X-Ray Explainer (ChEX). Evaluations across a heterogeneous set of 9 chest X-ray tasks, including localized image interpretation and report generation, showcase its competitiveness with SOTA models while additional analysis demonstrates ChEX’s interactive capabilities. Code: https://github.com/philip-mueller/chex.
Altmetric
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Conference contribution
Keywords Radiology Report Generation ; Vision-language Modeling
ISSN (print) / ISBN 0302-9743
e-ISSN 1611-3349
Conference Title 18th European Conference on Computer Vision, ECCV 2024
Conference Date 29 September - 4 October 2024
Conference Location Milan
Journal Lecture Notes in Computer Science
Quellenangaben Volume: 15079 LNCS, Issue: , Pages: 92-111 Article Number: , Supplement: ,
Publisher Springer
Publishing Place Berlin [u.a.]
Institute(s) Institute for Machine Learning in Biomed Imaging (IML)
Grants German Academic Exchange Service (DAAD) under the Kondrad Zuse School of Excellence for Reliable AI (RelAI)
Bavarian Collaborative Research Project PRIPREKI of the Free State of Bavaria Funding Programme "Artificial Intelligence -Data Science"
Medical Informatics Initiative as part of the PrivateAIM Project
German Ministry of Education and Research
Bavarian State Ministry for Science and the Arts under the Munich Centre for Machine Learning (MCML)
German Federal Ministry of Education and Research
ERC Grant Deep4MI