Home
Deutsch
Advanced Search
Browse by ...
Publication overview
Contact persons
Help
Postprint
DOI
 |
Open Access Green |
as soon as is submitted to ZB.
The pitfalls of sample selection: A case study on lung nodule classification.
In: (4th International Workshop on Predictive Intelligence in Medicine, PRIME 2021, 1 October 2021, Virtual, Online). Berlin [u.a.]: Springer, 2021. 201-211 (Lect. Notes Comput. Sc. ; 12928 LNCS)
Using publicly available data to determine the performance of methodological contributions is important as it facilitates reproducibility and allows scrutiny of the published results. In lung nodule classification, for example, many works report results on the publicly available LIDC dataset. In theory, this should allow a direct comparison of the performance of proposed methods and assess the impact of individual contributions. When analyzing seven recent works, however, we find that each employs a different data selection process, leading to largely varying total number of samples and ratios between benign and malignant cases. As each subset will have different characteristics with varying difficulty for classification, a direct comparison between the proposed methods is thus not always possible, nor fair. We study the particular effect of truthing when aggregating labels from multiple experts. We show that specific choices can have severe impact on the data distribution where it may be possible to achieve superior performance on one sample distribution but not on another. While we show that we can further improve on the state-of-the-art on one sample selection, we also find that on a more challenging sample selection, on the same database, the more advanced models underperform with respect to very simple baseline methods, highlighting that the selected data distribution may play an even more important role than the model architecture. This raises concerns about the validity of claimed methodological contributions. We believe the community should be aware of these pitfalls and make recommendations on how these can be avoided in future work.
Altmetric
Additional Metrics?
Edit extra informations
Login
Publication type
Article: Conference contribution
ISSN (print) / ISBN
0302-9743
e-ISSN
1611-3349
Conference Title
4th International Workshop on Predictive Intelligence in Medicine, PRIME 2021
Conference Date
1 October 2021
Conference Location
Virtual, Online
Quellenangaben
Volume: 12928 LNCS,
Pages: 201-211
Publisher
Springer
Publishing Place
Berlin [u.a.]
Non-patent literature
Publications
Institute(s)
Institute for Machine Learning in Biomed Imaging (IML)