PuSH - Publication Server of Helmholtz Zentrum München

Scholz, D.* ; Erdur, A.C.* ; Ehm, V.* ; Meyer-Baese, A.C.* ; Peeken, J.C. ; Rueckert, D.* ; Wiestler, B.*

MM-DINOv2: Adapting foundation models for multi-modal medical ímage analysis.

In: (28th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2025, 23-27 September 2025, Daejeon). Berlin [u.a.]: Springer, 2026. 320-330 (Lect. Notes Comput. Sc. ; 15967 LNCS)
DOI
Open Access Green as soon as Postprint is submitted to ZB.
Vision foundation models like DINOv2 demonstrate remarkable potential in medical imaging despite their origin in natural image domains. However, their design inherently works best for uni-modal image analysis, limiting their effectiveness for multi-modal imaging tasks that are common in many medical fields, such as neurology and oncology. While supervised models perform well in this setting, they fail to leverage unlabeled datasets and struggle with missing modalities—a frequent challenge in clinical settings. To bridge these gaps, we introduce MM-DINOv2, a novel and efficient framework that adapts the pre-trained vision foundation model DINOv2 for multi-modal medical imaging. Our approach incorporates multi-modal patch embeddings, enabling vision foundation models to effectively process multi-modal imaging data. To address missing modalities, we employ full-modality masking, which encourages the model to learn robust cross-modality relationships. Furthermore, we leverage semi-supervised learning to harness large unlabeled datasets, enhancing both the accuracy and reliability of medical predictions. We demonstrate our approach on glioma subtype classification from multi-sequence brain MRI, achieving a Matthews Correlation Coefficient (MCC) of 0.6 on an external test set, surpassing state-of-the-art supervised approaches by +11.1%. Beyond this specific application, our framework provides a scalable and robust blueprint for various multi-modal medical imaging problems effectively leveraging vision foundation models pre-trained on natural images while addressing real-world clinical challenges such as missing data and limited annotations (The code is publicly available at: https://github.com/daniel-scholz/mm-dinov2).
Altmetric
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Conference contribution
Keywords Dinov2 ; Multi-modal Mri ; Semi-supervised Learning
ISSN (print) / ISBN 0302-9743
e-ISSN 1611-3349
Conference Title 28th International Conference on Medical Image Computing and Computer Assisted Intervention, MICCAI 2025
Conference Date 23-27 September 2025
Conference Location Daejeon
Journal Lecture Notes in Computer Science
Quellenangaben Volume: 15967 LNCS, Issue: , Pages: 320-330 Article Number: , Supplement: ,
Publisher Springer
Publishing Place Berlin [u.a.]
Institute(s) Institute of Radiation Medicine (IRM)