PuSH - Publication Server of Helmholtz Zentrum München: Harnessing Orthogonality to Train Low-Rank Neural Networks.

Information

Clues to quality of journals

Open Access Policy of Helmholtz Association 2016

CC Licencences

Publication Metrics

Navigation

Home

Deutsch

EVA: Electronic Publishing

New EVA Application

Research

Advanced Search

Browse by ...

... HMGU-Authors/Consortia

... Organizational Structure

... Journal

... Publication Type

... Research Data

... Arbeitsgruppen

... Publication Year

Publication overview

Statistics

Statistics (last 5 years)

OA Publications

Publish

Submit Publication

Import publication from...

...EVA

Report missing publication

Highlights

Support & Contact

Contact persons

Help

Data protection

Helmholtz Open Science

JANE Journal Estimator

SHERPA/RoMEO

DOAJ

Export:

Text

Endnote (RIS) BIB

BibTeX

Coquelin, D.* ; Flügel, K.* ; Weiel, M.* ; Kiefer, N.* ; Debus, C.* ; Streit, A.* ; Götz, M.*

Harnessing Orthogonality to Train Low-Rank Neural Networks.

In:. Frontiers, 2024. 2106-2113 (Front. Artif. Intell. ; 392)

Publ. Version/Full Text

DOI

	Open Access Hybrid

Abstract
Metrics
Extra information

This study explores the learning dynamics of neural networks by analyzing the singular value decomposition (SVD) of their weights throughout training. Our investigation reveals that an orthogonal basis within each multidimensional weight's SVD representation stabilizes during training. Building upon this, we introduce Orthogonality-Informed Adaptive Low-Rank (OIALR) training, a novel training method exploiting the intrinsic orthogonality of neural networks. OIALR seamlessly integrates into existing training workflows with minimal accuracy loss, as demonstrated by benchmarking on various datasets and well-established network architectures. With appropriate hyperparameter tuning, OIALR can surpass conventional training setups, including those of state-of-the-art models.

Altmetric