Psychometrics in experimental psychology
A case for calibration
Bibliographic Data
| ID | 21641133 |
|---|---|
| Authors | Dominik R Bach (0000-0003-3717-2036, University of Bonn, corresponding author) |
| Year | 2024 |
| Volume | 31 |
| Issue | 4 |
| Pages | 1461-1470 |
| Publication date | 2024-08-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Psychonomic Bulletin & Review (JOURNAL) |
| Journal identifiers | ISSN: 1069-9384 • E-ISSN: 1531-5320 |
| Publisher | Springer Science and Business Media LLC (PUBLISHER) |
| DOI | 10.3758/s13423-023-02421-z |
| PMID | 38148472 |
| OpenAlex | W4390226486 |
| Language | EN |
| References cited | 27 |
Psychometrics is historically grounded in the study of individual differences. Consequently, common metrics such as quantitative validity and reliability require between-person variance in a psychological variable to be meaningful. Experimental psychology, in contrast, deals with variance between treatments, and experiments often strive to minimise within-group person variance. In this article, I ask whether and how psychometric evaluation can be performed in experimental psychology. A commonly used strategy is to harness between-person variance in the treatment effect. Using simulated data, I show that this approach can be misleading when between-person variance is low, and in the face of methods variance. I argue that this situation is common in experimental psychology, because low between-person variance is desirable, and because methods variance is no more problematic in experimental settings than any other source of between-person variance. By relating validity and reliability with the corresponding concepts in measurement science outside psychology, I show how experiment-based calibration can serve to compare the psychometric quality of different measurement methods in experimental psychology
Cognition · Cognitive psychology · Epistemology · Experimental psychology · Explained variation · Psychometrics · Statistics · Applied Psychology · Behavioral and Psychological Studies · Behavioral Health and Interventions · Clinical Psychology · Computer Science · Mathematics · Mental Health Research Topics · Psychology · Social Psychology · Artificial Intelligence
Test Theory
A psychometrics of individual differences in experimental tasks
Hierarchically Nested Covariance Structure Models for Multitrait-Multimethod Data
Power failure
Evaluating Structural Equation Models with Unobservable Variables and Measurement Error
False-Positive Psychology
On the reliability of behavioral measures of cognitive control
The reliability paradox
The two disciplines of scientific psychology
Antifragile
Convergent and discriminant validation by the multitrait-multimethod matrix
Calibrating the experimental measurement of psychological attributes
Construct validity in psychological tests
| Highly cited | No |
|---|