PuSH - Publication Server of Helmholtz Zentrum München

Oppelt, M.P.* ; Lang-Richter, N.R.* ; Eskofier, B.M.

LUMOS: Self-supervised Pretraining for Multimodal Task Load Estimation in Adaptive Instructional Systems.

In: (8th International Conference, Adaptive Instructional Systems, AIS 2026, held as Part of the 28th HCI International Conference, HCII 2026, 26-31 July 2026, Montreal). Berlin [u.a.]: Springer, 2026. 101-120 (Lect. Notes Comput. Sc. ; 16737 LNCS)
DOI
Accurate task load (TL) detection is essential for adaptive instructional systems (AIS) that tailor instruction to learner states, yet progress is hindered by costly data collection and scarce, noisy annotations. Recordings often contain substantial unlabeled segments, like rest periods, training phases, questionnaire completion, that supervised methods cannot exploit. We present LUMOS (Learning from Unlabeled Multimodal Observations in Task Load Detection Systems), a self-supervised learning framework that leverages these unlabeled multimodal physiological and behavioral signals to learn transferable representations for TL estimation. LUMOS employs discriminative self-supervised objectives, including MoCo, SimCLR, SwAV, and Barlow Twins, with modality-aware augmentations across electrocardiography, photoplethysmography, electrodermal activity, respiration, electromyography, skin temperature, pupil diameter, and eye movement data. We evaluate LUMOS on 96 sessions spanning n-back tests, driving simulation, and gaming tasks (Overcooked! 2, Hogwarts Legacy) using ConvNeXt, xLSTM, and Transformer backbones. MoCo pretrained models achieve the highest accuracy (0.79 ± 0.01), outperforming fully supervised baselines (0.72 ± 0.03) and demonstrating robust generalization across domains. Gains are most pronounced in low label regimes, where self-supervised initialization consistently outperforms training from scratch. Our results demonstrate that contrastive pretraining on unlabeled segments reduces annotation requirements and improves generalization, providing a practical foundation for TL-aware AIS applications.
Altmetric
Additional Metrics?
[➜Log in]
Edit extra informations Login
Publication type Article: Conference contribution
Keywords Contrastive Learning ; Generalization ; Multimodal Sensing ; Self-supervised Learning ; Task Load Detection
ISSN (print) / ISBN 0302-9743
e-ISSN 1611-3349
Conference Title 8th International Conference, Adaptive Instructional Systems, AIS 2026, held as Part of the 28th HCI International Conference, HCII 2026
Conference Date 26-31 July 2026
Conference Location Montreal
Journal Lecture Notes in Computer Science
Quellenangaben Volume: 16737 LNCS, Issue: , Pages: 101-120 Article Number: , Supplement: ,
Publisher Springer
Publishing Place Berlin [u.a.]
Institute(s) Institute of AI for Health (AIH)