PuSH - Publication Server of Helmholtz Zentrum München

Nasirigerdeh, R. ; Torkzadehmahani, J.* ; Rueckert, D.* ; Kaissis, G.

Kernel Normalized Convolutional Networks for Privacy-Preserving Machine Learning.

In: (Proceedings - 2023 IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2023). 10662 Los Vaqueros Circle, Po Box 3014, Los Alamitos, Ca 90720-1264 Usa: Ieee Computer Soc, 2023. 107-118 (Proceedings - 2023 IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2023)
DOI
Normalization is an important but understudied challenge in privacy-related application domains such as federated learning (FL), differential privacy (DP), and differentially private federated learning (DP-FL). While the unsuitability of batch normalization for these domains has already been shown, the impact of other normalization methods on the performance of federated or differentially private models is not well-known. To address this, we draw a performance comparison among layer normalization (LayerNorm), group normalization (GroupNorm), and the recently proposed kernel normalization (KernelNorm) in FL, DP, and DP-FL settings. Our results indicate LayerNorm and GroupNorm provide no performance gain compared to the baseline (i.e. no normalization) for shallow models in FL and DP. They, on the other hand, considerably enhance the performance of shallow models in DP-FL and deeper models in FL and DP. KernelNorm, moreover, significantly outperforms its competitors in terms of accuracy and convergence rate (or communication efficiency) for both shallow and deeper models in all considered learning environments. Given these key observations, we propose a kernel normalized ResNet architecture called KNResNet-13 for differentially private learning. Using the proposed architecture, we provide new state-of-the-art accuracy values on the CIFAR-10 and Imagenette datasets, when trained from scratch.
Altmetric
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Conference contribution
Corresponding Author
Keywords Batch Normalization ; Differential Privacy ; Federated Learning ; Group Normalization ; Kernel Normalization
ISSN (print) / ISBN 9781665462990
Conference Title Proceedings - 2023 IEEE Conference on Secure and Trustworthy Machine Learning, SaTML 2023
Quellenangaben Volume: , Issue: , Pages: 107-118 Article Number: , Supplement: ,
Publisher Ieee Computer Soc
Publishing Place 10662 Los Vaqueros Circle, Po Box 3014, Los Alamitos, Ca 90720-1264 Usa
Non-patent literature Publications
Institute(s) Helmholtz Artifical Intelligence Cooperation Unit (HAICU)
Institute for Machine Learning in Biomed Imaging (IML)