PuSH - Publikationsserver des Helmholtz Zentrums München

Erdur, A.C.* ; Beischl, C.* ; Scholz, D.* ; Pan, J.* ; Wiestler, B.* ; Rueckert, D.* ; Peeken, J.C.

MultiMAE for Brain MRIs: Robustness to Missing Inputs Using Multi-modal Masked Autoencoder.

In:. Berlin [u.a.]: Springer, 2026. 572 - 582 (Lect. Notes Comput. Sc. ; 16241 LNCS)
DOI
Missing input sequences are common in medical imaging data, posing a challenge for deep learning models reliant on complete input data. In this work, inspired by MultiMAE [2], we develop a masked autoencoder (MAE) paradigm for multi-modal, multi-task learning in 3D medical imaging with brain MRIs. Our method treats each MRI sequence as a separate input modality, leveraging a late-fusion-style transformer encoder to integrate multi-sequence information (“multi-modal”) and individual decoder streams for each modality for “multi-task” reconstruction. This pretraining strategy guides the model to learn rich representations per modality while also equipping it to handle missing inputs through cross-sequence reasoning. The result is a flexible and generalizable encoder for brain MRIs that infers missing sequences from available inputs and can be adapted to various downstream applications. We demonstrate the performance and robustness of our method against an MAE-ViT baseline in downstream segmentation and classification tasks, showing absolute improvement of 10.1 overall Dice score and 0.46 MCC over the baselines with missing input sequences. Our experiments demonstrate the strength of this pretraining strategy. The implementation is made available (The repository for our work: https://github.com/chris-beischl/multimae-for-brain-mri).
Altmetric
Weitere Metriken?
[➜Einloggen]
Zusatzinfos bearbeiten [➜Einloggen]
Publikationstyp Artikel: Sammelbandbeitrag/Buchkapitel
Schlagwörter Brain Mri ; Masked Autoencoder Pretraining ; Missing Modalities
ISSN (print) / ISBN 0302-9743
e-ISSN 1611-3349
Zeitschrift Lecture Notes in Computer Science
Quellenangaben Band: 16241 LNCS, Heft: , Seiten: 572 - 582 Artikelnummer: , Supplement: ,
Verlag Springer
Verlagsort Berlin [u.a.]
Institut(e) Institute of Radiation Medicine (IRM)