PuSH - Publication Server of Helmholtz Zentrum München: Architectural insights into and training methodology optimization of Pangu-Weather.

Information

Clues to quality of journals

Open Access Policy of Helmholtz Association 2016

CC Licencences

Publication Metrics

Navigation

Home

Deutsch

EVA: Electronic Publishing

New EVA Application

Research

Advanced Search

Browse by ...

... HMGU-Authors/Consortia

... Organizational Structure

... Journal

... Publication Type

... Research Data

... Arbeitsgruppen

... Publication Year

Publication overview

Statistics

Statistics (last 5 years)

OA Publications

Publish

Submit Publication

Import publication from...

...EVA

Report missing publication

Highlights

Support & Contact

Contact persons

Help

Data protection

Helmholtz Open Science

JANE Journal Estimator

SHERPA/RoMEO

DOAJ

Export:

Text

Endnote (RIS) BIB

BibTeX

To, D.* ; Quinting, J.* ; Hoshyaripour, G.A.* ; Goetz, M.* ; Streit, A.* ; Debus, C.*

Architectural insights into and training methodology optimization of Pangu-Weather.

Geosci. Model. Dev. 17, 8873-8884 (2024)

Publ. Version/Full Text

DOI

	Open Access Gold

Abstract
Metrics
Extra information

Data-driven medium-range weather forecasts have recently outperformed classical numerical weather prediction models, with Pangu-Weather (PGW) being the first breakthrough model to achieve this. The Transformer-based PGW introduced novel architectural components including the three-dimensional attention mechanism (3D Transformer) in the Transformer blocks. Additionally, it features an Earth-specific positional bias term which accounts for weather states being related to the absolute position on Earth. However, the effectiveness of different architectural components is not yet well understood. Here, we reproduce the 24 h forecast model of PGW based on subsampled 6-hourly data. We then present an ablation study of PGW to better understand the sensitivity to the model architecture and training procedure. We find that using a two-dimensional attention mechanism (2D Transformer) yields a model that is more robust to training, converges faster, and produces better forecasts compared to using the 3D Transformer. The 2D Transformer reduces the overall computational requirements by 20 %-30 %. Further, the Earth-specific positional bias term can be replaced with a relative bias, reducing the model size by nearly 40 %. A sensitivity study comparing the convergence of the PGW model and the 2D-Transformer model shows large batch effects; however, the 2D-Transformer model is more robust to such effects. Lastly, we propose a new training procedure that increases the speed of convergence for the 2D-Transformer model by 30 % without any further hyperparameter tuning.

Impact Factor

Scopus SNIP

Altmetric

4.000

0.000