PuSH - Publication Server of Helmholtz Zentrum München

Zhou, Y.* ; Fritz, M.* ; Keuper, M.

MultiMax: Sparse and multi-modal attention learning.

In: (41st International Conference on Machine Learning, 21-27 July 2024, Vienna). 2024. 61897-61912 (Proceedings of Machine Learning Research ; 235)
SoftMax is a ubiquitous ingredient of modern machine learning algorithms.It maps an input vector onto a probability simplex and reweights the input by concentrating the probability mass at large entries.Yet, as a smooth approximation to the Argmax function, a significant amount of probability mass is distributed to other, residual entries, leading to poor interpretability and noise.Although sparsity can be achieved by a family of SoftMax variants, they often require an alternative loss function and do not preserve multi-modality.We show that this trade-off between multi-modality and sparsity limits the expressivity of SoftMax as well as its variants.We provide a solution to this tension between objectives by proposing a piece-wise differentiable function, termed MultiMax, which adaptively modulates the output distribution according to input entry range.Through comprehensive analysis and evaluation, we show that MultiMax successfully produces a distribution that supresses irrelevant entries while preserving multi-modality, with benefits in image classification, language modeling and machine translation.The code is available at https://github.com/ZhouYuxuanYX/MultiMax.
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Conference contribution
Corresponding Author
Conference Title 41st International Conference on Machine Learning
Conference Date 21-27 July 2024
Conference Location Vienna
Quellenangaben Volume: 235, Issue: , Pages: 61897-61912 Article Number: , Supplement: ,
Non-patent literature Publications
Institute(s) Institute of Diabetes Research and Metabolic Diseases (IDM)