PuSH - Publikationsserver des Helmholtz Zentrums München

Voudouris, K. ; Thalmann, M. ; Kipnis, A. ; Hernández-Orallo, J.* ; Schulz, E.

Measuring what AI systems might do: Towards a measurement science in AI.

Minds Mach. 36:50 (2026)
Verlagsversion DOI
Open Access Hybrid
Creative Commons Lizenzvertrag
Scientists, policy-makers, business leaders, and members of the public care about what modern artificial intelligence systems are disposed to do. Yet terms such as capabilities, propensities, skills, values, and abilities are routinely used interchangeably and conflated with observable performance, with AI evaluation practices rarely specifying what quantity they purport to measure. Building on established dispositional accounts of ability, we argue that capabilities and propensities are dispositional properties—stable features of systems characterised by counterfactual relationships between contextual conditions and behavioural outputs. Measuring a disposition requires (i) hypothesising which contextual properties are causally relevant, (ii) independently operationalising and measuring those properties, and (iii) empirically mapping how variation in those properties affects the probability of the behaviour. Benchmark averages and data-driven latent-variable models like Item Response Theory can provide useful evidence, but do not by themselves identify these causal structures. By developing this principled account of capabilities, propensities, and how to measure them, we are able to outline what disposition-respecting, scientifically defensible AI evaluation would require.
Altmetric
Weitere Metriken?
Zusatzinfos bearbeiten [➜Einloggen]
Publikationstyp Artikel: Journalartikel
Dokumenttyp Review
Schlagwörter Ai Evaluation ; Measurement Theory ; Dispositions ; Ai Capabilities ; Dispositional Properties ; Counterfactual Reasoning ; Causal Relevance ; Item Response Theory ; Philosophy Of Science ; Metrology
ISSN (print) / ISBN 0924-6495
e-ISSN 1572-8641
Zeitschrift Minds and Machines
Quellenangaben Band: 36, Heft: 4, Seiten: , Artikelnummer: 50 Supplement: ,
Verlag Springer
Verlagsort Van Godewijckstraat 30, 3311 Gz Dordrecht, Netherlands
Begutachtungsstatus Peer reviewed
Förderungen Helmholtz Zentrum Mnchen - Deutsches Forschungszentrum fr Gesundheit und Umwelt (GmbH) (4209)