PuSH - Publication Server of Helmholtz Zentrum München

Ruppik, B.M.* ; von Rohrscheidt, J.C. ; van Niekerk, C.* ; Heck, M.* ; Vukovic, R.* ; Feng, S.* ; Lin, H.C.* ; Lubis, N.* ; Rieck, B. ; Zibrowius, M.* ; Gašić, M.*

Less is More: Local Intrinsic Dimensions of Contextual Language Models.

In: (39th Conference on Neural Information Processing Systems, NeurIPS 2025, 02-07 December 2025, San Diego). 2025. 75559-75592 (Advances in Neural Information Processing Systems ; 38)
Understanding the internal mechanisms of large language models (LLMs) remains a challenging and complex endeavor. Even fundamental questions, such as how fine-tuning affects model behavior, often require extensive empirical evaluation. In this paper, we introduce a novel perspective based on the geometric properties of contextual latent embeddings to study the effects of training and fine-tuning. To that end, we measure the local dimensions of a contextual language model's latent space and analyze their shifts during training and fine-tuning. We show that the local dimensions provide insights into the model's training dynamics and generalization ability. Specifically, the mean of the local dimensions predicts when the model's training capabilities are exhausted, as exemplified in a dialogue state tracking task, overfitting, as demonstrated in an emotion recognition task, and grokking, as illustrated with an arithmetic task. Furthermore, our experiments suggest a practical heuristic: reductions in the mean local dimension tend to accompany and predict subsequent performance gains. Through this exploration, we aim to provide practitioners with a deeper understanding of the implications of fine-tuning on embedding spaces, facilitating informed decisions when configuring models for specific applications. The results of this work contribute to the ongoing discourse on the interpretability, adaptability, and generalizability of LLMs by bridging the gap between intrinsic model mechanisms and geometric properties in embeddings.
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Conference contribution
ISSN (print) / ISBN 1049-5258
Conference Title 39th Conference on Neural Information Processing Systems, NeurIPS 2025
Conference Date 02-07 December 2025
Conference Location San Diego
Quellenangaben Volume: 38, Issue: , Pages: 75559-75592 Article Number: , Supplement: ,
Institute(s) Institute of AI for Health (AIH)