PuSH - Publication Server of Helmholtz Zentrum München

Consens, M.E.* ; Dufault, C.* ; Wainberg, M.* ; Forster, D.* ; Karimzadeh, M.* ; Goodarzi, H.* ; Theis, F.J. ; Moses, A.* ; Wang, B.*

Transformers and genome language models.

Nat. Mach. Intell., DOI: 10.1038/s42256-025-01007-9 (2025)
DOI
Large language models based on the transformer deep learning architecture have revolutionized natural language processing. Motivated by the analogy between human language and the genome’s biological code, researchers have begun to develop genome language models (gLMs) based on transformers and related architectures. This Review explores the use of transformers and language models in genomics. We survey open questions in genomics amenable to the use of gLMs, and motivate the use of gLMs and the transformer architecture for these problems. We discuss the potential of gLMs for modelling the genome using unsupervised pretraining tasks, specifically focusing on the power of zero- and few-shot learning. We explore the strengths and limitations of the transformer architecture, as well as the strengths and limitations of current gLMs more broadly. Additionally, we contemplate the future of genomic modelling beyond the transformer architecture, based on current trends in research. This Review serves as a guide for computational biologists and computer scientists interested in transformers and language models for genomic data.
Altmetric
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Journal article
Document type Review
Corresponding Author
ISSN (print) / ISBN 2522-5839
e-ISSN 2522-5839
Journal Nature machine intelligence
Publisher Springer
Publishing Place [London]
Non-patent literature Publications
Reviewing status Peer reviewed
Institute(s) Institute of Computational Biology (ICB)