PuSH - Publication Server of Helmholtz Zentrum München

Grashei, C.* ; Brechenmacher, C. ; Umer, R.M. ; Liu, J.* ; Marr, C. ; Schüffler, P.* ; Szczurek, E.

Pathryoshka: Compressing Pathology Foundation Models via Multi-teacher Knowledge Distillation with Nested Embeddings.

In: (Computer Vision – ECCV 2026). Berlin [u.a.]: Springer, 2026. 76-94 (Lect. Notes Comput. Sc. ; 17074 LNCS)
DOI
Foundation models (FMs) have driven significant progress in computational pathology. These models can easily exceed a billion parameters and produce high–dimensional embeddings, thus limiting their applicability for research or clinical use when computing resources are tight. Here we introduce Pathryoshka, a novel multi–teacher distillation framework inspired by agglomerative models and Matryoshka representation learning to reduce pathology FM sizes while allowing for adaptable embedding dimensions. We evaluate our framework with a distilled model on ten public pathology benchmarks with varying downstream tasks. Compared to its much larger teachers, Pathryoshka reduces the model size by 86–92% at on–par performance. It outperforms state–of–the–art single–teacher distillation models of comparable size by a median margin of 7.0 and other pathology multi–teacher distillation by 5.3 percentage points in accuracy. By enabling efficient deployment without sacrificing accuracy or representational richness, Pathryoshka democratizes access to state–of–the–art pathology FMs for the broader research and clinical community.
Altmetric
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Conference contribution
Keywords Computational Pathology ; Foundation Model ; Knowledge Distillation
ISSN (print) / ISBN 0302-9743
e-ISSN 1611-3349
Conference Title Computer Vision – ECCV 2026
Journal Lecture Notes in Computer Science
Quellenangaben Volume: 17074 LNCS, Issue: , Pages: 76-94 Article Number: , Supplement: ,
Publisher Springer
Publishing Place Berlin [u.a.]
Institute(s) Institute of AI for Health (AIH)