PuSH - Publication Server of Helmholtz Zentrum München

Traoré, K.R.* ; Camero, A.* ; Zhu, X.X.*

We Won’t Get Fooled Again: When Performance Metric Malfunction Affects the Landscape of Hyperparameter Optimization Problems.

In:. Springer, 2023. 148-160 (Comm. Comp. Info. Sci. ; 1824 CCIS)
DOI
Hyperparameter optimization (HPO) is a well-studied research field. However, the effects and interactions of the components in an HPO pipeline are not yet well investigated. Then, we ask ourselves: Can the landscape of HPO be biased by the pipeline used to evaluate individual configurations? To address this question, we proposed to analyze the effect of the HPO pipeline on HPO problems using fitness landscape analysis. Particularly, we studied over 119 generic classification instances from either the DS-2019 (CNN) and YAHPO (XGBoost) HPO benchmark data sets, looking for patterns that could indicate evaluation pipeline malfunction, and relate them to HPO performance. Our main findings are: (i) In most instances, large groups of diverse hyperparameters (i.e., multiple configurations) yield the same ill performance, most likely associated with majority class prediction models (predictive accuracy) or models unable to attribute an appropriate class to observations (log loss); (ii) in these cases, a worsened correlation between the observed fitness and average fitness in the neighborhood is observed, potentially making harder the deployment of local-search-based HPO strategies. (iii) these effects are observed across different HPO scenarios (tuning CNN or XGBoost algorithms). Finally, we concluded that the HPO pipeline definition might negatively affect the HPO landscape.
Altmetric
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Conference contribution
Keywords Benchmarking ; Fitness Landscape Analysis ; Hyperparameter Optimization
ISSN (print) / ISBN 1865-0929
e-ISSN 1865-0937
Journal Communications in Computer and Information Science
Quellenangaben Volume: 1824 CCIS, Issue: , Pages: 148-160 Article Number: , Supplement: ,
Publisher Springer
Reviewing status Peer reviewed
Institute(s) Helmholtz AI - DLR (HAI - DLR)