PuSH - Publication Server of Helmholtz Zentrum München

Maier-Hein, L.* ; Reinke, A.* ; Godau, P.* ; Tizabi, M.D.* ; Buettner, F.* ; Christodoulou, E.* ; Glocker, B.* ; Isensee, F.* ; Kleesiek, J.* ; Kozubek, M.* ; Reyes, M.* ; Riegler, M.A.* ; Wiesenfarth, M.* ; Kavur, A.E.* ; Sudre, C.H.* ; Baumgartner, M.* ; Eisenmann, M.* ; Heckmann-Nötzel, D.* ; Rädsch, T.* ; Acion, L.* ; Antonelli, M.* ; Arbel, T.* ; Bakas, S.* ; Benis, A.* ; Blaschko, M.B.* ; Cardoso, M.J.* ; Cheplygina, V.* ; Cimini, B.A.* ; Collins, G.S.* ; Farahani, K.* ; Ferrer, L.* ; Galdran, A.* ; Van Ginneken, B.* ; Haase, R.* ; Hashimoto, D.A.* ; Hoffman, M.M.* ; Huisman, M.* ; Jannin, P.* ; Kahn, C.E.* ; Kainmueller, D.* ; Kainz, B.* ; Karargyris, A.* ; Karthikesalingam, A.* ; Kofler, F. ; Kopp-Schneider, A.* ; Kreshuk, A.* ; Kurc, T.* ; Landman, B.A.* ; Litjens, G.* ; Madani, A.* ; Maier-Hein, K.* ; Martel, A.L.* ; Mattson, P.* ; Meijering, E.* ; Menze, B.* ; Moons, K.G.M.* ; Müller, H.* ; Nichyporuk, B.* ; Nickel, F.* ; Petersen, J.* ; Rajpoot, N.* ; Rieke, N.* ; Saez-Rodriguez, J.* ; Sánchez, C.I.* ; Shetty, S.* ; van Smeden, M.* ; Summers, R.M.* ; Taha, A.A.* ; Tiulpin, A.* ; Tsaftaris, S.A.* ; van Calster, B.* ; Varoquaux, G.* ; Jäger, P.F.*

Metrics reloaded: Recommendations for image analysis validation.

Nat. Methods 21, 195-212 (2024)
DOI
Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint—a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.
Altmetric
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Journal article
Document type Review
Corresponding Author
Keywords Health; Segmentation; Criteria
ISSN (print) / ISBN 1548-7091
e-ISSN 1548-7105
Journal Nature Methods
Quellenangaben Volume: 21, Issue: 2, Pages: 195-212 Article Number: , Supplement: ,
Publisher Nature Publishing Group
Publishing Place New York, NY
Non-patent literature Publications
Reviewing status Peer reviewed
Institute(s) Helmholtz Artifical Intelligence Cooperation Unit (HAICU)
Grants Ministry of Education, Youth and Sports of the Czech Republic
French State Funds (IHU Strasbourg)
Natural Sciences and Engineering Research Council of Canada
Cancer Research UK
Chan Zuckerberg Initiative DAF
NIH
Independent Research Council Denmark
Novo Nordisk Foundation
European Union (ERC)
National Institute of Neurological Disorders and Stroke (NINDS) of the NIH
National Cancer Institute (NCI)
Intramural Research Program of the National Institutes of Health (NIH) Clinical Center
Dutch Research Council
Dutch Cancer Association

Research Chairs and Senior Research Fellowships scheme
Canon Medical and the Royal Academy of Engineering
Terttu foundation
Wellbeing Services County of North Ostrobothnia
Finnish Foundation for Cardiovascular Research
University of Oulu
Academy of Finland
Intramural Research Program of the NIH Clinical Center
Swiss National Science Foundation
Innosuisse
Alzheimer's Society Junior Fellowship
Innovative Medicine Initiative
European Union
ERC
HealthHolland
European Research Council (ERC) under the European Union