PuSH - Publikationsserver des Helmholtz Zentrums München

Zhou, Y.* ; Fritz, M.* ; Keuper, M.

MultiMax: Sparse and multi-modal attention learning.

In: (41st International Conference on Machine Learning, 21-27 July 2024, Vienna). 2024. 61897-61912 (Proceedings of Machine Learning Research ; 235)
SoftMax is a ubiquitous ingredient of modern machine learning algorithms.It maps an input vector onto a probability simplex and reweights the input by concentrating the probability mass at large entries.Yet, as a smooth approximation to the Argmax function, a significant amount of probability mass is distributed to other, residual entries, leading to poor interpretability and noise.Although sparsity can be achieved by a family of SoftMax variants, they often require an alternative loss function and do not preserve multi-modality.We show that this trade-off between multi-modality and sparsity limits the expressivity of SoftMax as well as its variants.We provide a solution to this tension between objectives by proposing a piece-wise differentiable function, termed MultiMax, which adaptively modulates the output distribution according to input entry range.Through comprehensive analysis and evaluation, we show that MultiMax successfully produces a distribution that supresses irrelevant entries while preserving multi-modality, with benefits in image classification, language modeling and machine translation.The code is available at https://github.com/ZhouYuxuanYX/MultiMax.
Weitere Metriken?
[➜Einloggen]
Zusatzinfos bearbeiten [➜Einloggen]
Publikationstyp Artikel: Konferenzbeitrag
Konferenztitel 41st International Conference on Machine Learning
Konferzenzdatum 21-27 July 2024
Konferenzort Vienna
Quellenangaben Band: 235, Heft: , Seiten: 61897-61912 Artikelnummer: , Supplement: ,
Institut(e) Institute of Diabetes Research and Metabolic Diseases (IDM)