PuSH - Publication Server of Helmholtz Zentrum München

Bercea, C.I.* ; Li, J.* ; Raffler, P.* ; Riedel, E.O.* ; Schmitzer, L.* ; Kurz, A.* ; Bitzer, F.* ; Roßmüller, P.* ; Canisius, J.* ; Beyrle, M.L.* ; Liu, C.* ; Bai, W.* ; Kainz, B.* ; Schnabel, J.A. ; Wiestler, B.*

NOVA: A Benchmark for Rare Anomaly Localization and Clinical Reasoning in Brain MRI.

In: (39th Conference on Neural Information Processing Systems, NeurIPS 2025, 02-07 December 2025, San Diego). 2025. 115600-115633 (Advances in Neural Information Processing Systems ; 38)
In many real-world applications, deployed models encounter inputs that differ from the data seen during training. Open-world recognition ensures that such systems remain robust as ever-emerging, previously unknown categories appear and must be addressed without retraining. Foundation and vision-language models are pretrained on large and diverse datasets with the expectation of broad generalization across domains, including medical imaging. However, benchmarking these models on test sets with only a few common outlier types silently collapses the evaluation back to a closed-set problem, masking failures on rare or truly novel conditions encountered in clinical use. We therefore present NOVA, a challenging, real-life evaluation-only benchmark of ∼900 brain MRI scans that span 281 rare pathologies and heterogeneous acquisition protocols. Each case includes rich clinical narratives and double-blinded expert bounding-box annotations. Together, these enable joint assessment of anomaly localisation, visual captioning, and diagnostic reasoning. Because NOVA is never used for training, it serves as an extreme stress-test of out-of-distribution generalisation: models must bridge a distribution gap both in sample appearance and in semantic space. Baseline results with leading vision-language models (GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL-72B) reveal substantial performance drops, with approximately a 65% gap in localisation compared to natural-image benchmarks and 40% and 20% gaps in captioning and reasoning, respectively, compared to resident radiologists. Therefore, NOVA establishes a testbed for advancing models that can detect, localize, and reason about truly unknown anomalies.
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Conference contribution
ISSN (print) / ISBN 1049-5258
Conference Title 39th Conference on Neural Information Processing Systems, NeurIPS 2025
Conference Date 02-07 December 2025
Conference Location San Diego
Quellenangaben Volume: 38, Issue: , Pages: 115600-115633 Article Number: , Supplement: ,
Institute(s) Institute of Machine Learning in Biomedical Imaging (IML)