PuSH - Publication Server of Helmholtz Zentrum München

Tetko, I.V. ; Van Deursen, R.* ; Godin, G.*

Be aware of overfitting by hyperparameter optimization!

J. Cheminformatics 16:139 (2024)
Publ. Version/Full Text DOI PMC
Open Access Gold
Creative Commons Lizenzvertrag
Hyperparameter optimization is very frequently employed in machine learning. However, an optimization of a large space of parameters could result in overfitting of models. In recent studies on solubility prediction the authors collected seven thermodynamic and kinetic solubility datasets from different data sources. They used state-of-the-art graph-based methods and compared models developed for each dataset using different data cleaning protocols and hyperparameter optimization. In our study we showed that hyperparameter optimization did not always result in better models, possibly due to overfitting when using the same statistical measures. Similar results could be calculated using pre-set hyperparameters, reducing the computational effort by around 10,000 times. We also extended the previous analysis by adding a representation learning method based on Natural Language Processing of smiles called Transformer CNN. We show that across all analyzed sets using exactly the same protocol, Transformer CNN provided better results than graph-based methods for 26 out of 28 pairwise comparisons by using only a tiny fraction of time as compared to other methods. Last but not least we stressed the importance of comparing calculation results using exactly the same statistical measures.Scientific Contribution We showed that models with pre-optimized hyperparameters can suffer from overfitting and that using pre-set hyperparameters yields similar performances but four orders faster. Transformer CNN provided significantly higher accuracy compared to other investigated methods.
Altmetric
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Journal article
Document type Scientific Article
Corresponding Author
Keywords Aqueous Solubility
e-ISSN 1758-2946
Journal Journal of Cheminformatics
Quellenangaben Volume: 16, Issue: 1, Pages: , Article Number: 139 Supplement: ,
Publisher BioMed Central
Publishing Place Campus, 4 Crinan St, London N1 9xw, England
Non-patent literature Publications
Reviewing status Peer reviewed
Institute(s) Institute of Structural Biology (STB)
Grants European Union
Projekt DEAL