PuSH - Publication Server of Helmholtz Zentrum München: CogBench: A large language model walks into a psychology lab.

Information

Clues to quality of journals

Open Access Policy of Helmholtz Association 2016

CC Licencences

Publication Metrics

Navigation

Home

Deutsch

Research

Advanced Search

Browse by ...

... HMGU-Authors/Consortia

... Organizational Structure

... Journal

... Publication Type

... Research Data

... Arbeitsgruppen

... Publication Year

Publication overview

Statistics

Statistics (last 5 years)

OA Publications

Publish

Submit Publication

Import publication from...

...EVA

Report missing publication

Highlights

Support & Contact

Contact persons

Help

Data protection

Helmholtz Open Science

JANE Journal Estimator

SHERPA/RoMEO

DOAJ

Export:

Text

Endnote (RIS) BIB

BibTeX

Coda-Forno, J. ; Binz, M. ; Wang, J.X.* ; Schulz, E.

CogBench: A large language model walks into a psychology lab.

In: (Proceedings of Machine Learning Research). 2024. 9076-9108 (Proceedings of Machine Learning Research ; 235)

Abstract
Metrics
Extra information

Large language models (LLMs) have significantly advanced the field of artificial intelligence. Yet, evaluating them comprehensively remains challenging. We argue that this is partly due to the predominant focus on performance metrics in most benchmarks. This paper introduces CogBench, a benchmark that includes ten behavioral metrics derived from seven cognitive psychology experiments. This novel approach offers a toolkit for phenotyping LLMs' behavior. We apply CogBench to 40 LLMs, yielding a rich and diverse dataset. We analyze this data using statistical multilevel modeling techniques, accounting for the nested dependencies among fine-tuned versions of specific LLMs. Our study highlights the crucial role of model size and reinforcement learning from human feedback (RLHF) in improving performance and aligning with human behavior. Interestingly, we find that open-source models are less risk-prone than proprietary models and that fine-tuning on code does not necessarily enhance LLMs' behavior. Finally, we explore the effects of prompt-engineering techniques. We discover that chain-of-thought prompting improves probabilistic reasoning, while take-a-step-back prompting fosters model-based behaviors.