PuSH - Publication Server of Helmholtz Zentrum München

Export: TXT Text RIS Endnote (RIS) BIB BibTeX
Sadeghi, M.* ; Richer, R.* ; Egger, B.* ; Schindler-Gmelch, L.* ; Rupp, L.H.* ; Rahimi, F.* ; Berking, M.* ; Eskofier, B.M.

Harnessing multimodal approaches for depression detection using large language models and facial expressions.

Npj Ment. Health Res. 3:66 (2024)
Publ. Version/Full Text DOI PMC
Free journal
Creative Commons Lizenzvertrag
Detecting depression is a critical component of mental health diagnosis, and accurate assessment is essential for effective treatment. This study introduces a novel, fully automated approach to predicting depression severity using the E-DAIC dataset. We employ Large Language Models (LLMs) to extract depression-related indicators from interview transcripts, utilizing the Patient Health Questionnaire-8 (PHQ-8) score to train the prediction model. Additionally, facial data extracted from video frames is integrated with textual data to create a multimodal model for depression severity prediction. We evaluate three approaches: text-based features, facial features, and a combination of both. Our findings show the best results are achieved by enhancing text data with speech quality assessment, with a mean absolute error of 2.85 and root mean square error of 4.02. This study underscores the potential of automated depression detection, showing text-only models as robust and effective while paving the way for multimodal analysis.
Impact Factor
Scopus SNIP
Web of Science
Times Cited
Altmetric
0.000
0.000
8
Tags
Annotations
Special Publikation
Hide on homepage

Edit extra information
Edit own tags
Private
Edit own annotation
Private
Hide on publication lists
on hompage
Mark as special
publikation
Publication type Article: Journal article
Document type Scientific Article
Language english
Publication Year 2024
HGF-reported in Year 2024
ISSN (print) / ISBN 2731-4251
e-ISSN 2731-4251
Journal Npj Mental Health Research
Quellenangaben Volume: 3, Issue: 1, Pages: , Article Number: 66 Supplement: ,
Publisher Springer
Publishing Place Campus, 4 Crinan St, London, N1 9xw, England
Reviewing status Peer reviewed
Institute(s) Institute of AI for Health (AIH)
POF-Topic(s) 30205 - Bioengineering and Digital Health
Research field(s) Enabling and Novel Technologies
PSP Element(s) G-540008-001
Grants Deutsche Forschungsgemeinschaft (DFG, German Research foundation)
DOI 10.1038/s44184-024-00112-8
WOS ID 001592604200001
Scopus ID 105004858671
PubMed ID 39715786
Erfassungsdatum 2025-01-09
[➜Report correction]