PuSH - Publikationsserver des Helmholtz Zentrums München: MultiMax: Sparse and multi-modal attention learning.

Informationen

Hinweise zu Qualitätskriterien von Zeitschriften

Open-Access-Richtlinie der Helmholtz-Gemeinschaft 2016

CC Licencences

Metriken für Publikationen

Navigation

Startseite

English

EVA: Veröffentlichen

Neuer EVA Antrag

Recherche

Erweiterte Suche

Durchblättern nach ...

... HMGU-Autoren/Konsortien

... Organisationsstruktur

... Zeitschriften

... Publikationstypen

... Forschungsdaten

... Arbeitsgruppen

... Erscheinungsjahr

Publikationen im Überblick

Statistik

HGF Fortschrittsbericht

OA Publikationen

Eintragen

Neue Publikation eintragen

Neue Publikation holen aus...

...EVA

Fehlende Publikation melden

Highlights

Suche

Hilfe & Kontakt

Ansprechpartner

Hilfe

Datenschutz

Helmholtz Open Science

Bibliometrische Indikatoren

SHERPA/RoMEO

DOAJ

Export:

Text

Endnote (RIS) BIB

BibTeX

Zhou, Y.* ; Fritz, M.* ; Keuper, M.

MultiMax: Sparse and multi-modal attention learning.

In: (41st International Conference on Machine Learning, 21-27 July 2024, Vienna). 2024. 61897-61912 (Proceedings of Machine Learning Research ; 235)

Abstract
Metriken
Zusatzinfos

SoftMax is a ubiquitous ingredient of modern machine learning algorithms.It maps an input vector onto a probability simplex and reweights the input by concentrating the probability mass at large entries.Yet, as a smooth approximation to the Argmax function, a significant amount of probability mass is distributed to other, residual entries, leading to poor interpretability and noise.Although sparsity can be achieved by a family of SoftMax variants, they often require an alternative loss function and do not preserve multi-modality.We show that this trade-off between multi-modality and sparsity limits the expressivity of SoftMax as well as its variants.We provide a solution to this tension between objectives by proposing a piece-wise differentiable function, termed MultiMax, which adaptively modulates the output distribution according to input entry range.Through comprehensive analysis and evaluation, we show that MultiMax successfully produces a distribution that supresses irrelevant entries while preserving multi-modality, with benefits in image classification, language modeling and machine translation.The code is available at https://github.com/ZhouYuxuanYX/MultiMax.