A Framework of Construct-Irrelevant Variance for Contextualized Constructed Response Assessment
Bibliographic Data
| ID | 22166162 |
|---|---|
| Authors | Xiaoming Zhai (0000-0003-4519-1931, University of Georgia, corresponding author), Kevin C Haudek (0000-0003-1422-6038, Michigan State University), Christopher Wilson (0000-0002-0218-6895), Christopher D Wilson (0000-0002-1628-5106, BSCS Science Learning), Molly A M Stuhlsatz (0000-0001-7218-1240, BSCS Science Learning), Molly Stuhlsatz |
| Year | 2021 |
| Volume | 6 |
| Publication date | 2021-10-12 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Frontiers in Education (JOURNAL) |
| Journal identifiers | ISSN: 2504-284X • E-ISSN: 2504-284X |
| Publisher | Frontiers Media SA (PUBLISHER • CH) |
| DOI | 10.3389/feduc.2021.751283 |
| OpenAlex | W3205205007 |
| Language | EN |
| Citations received | 4 |
| References cited | 47 |
Estimating and monitoring the construct-irrelevant variance (CIV) is of significant importance to validity, especially for constructed response assessments with rich contextualized information. To examine CIV in contextualized constructed response assessments, we developed a framework including a model accounting for CIV and a measurement that could differentiate the CIV. Specifically, the model includes CIV due to three factors: the variability of assessment item scenarios, judging severity, and rater scoring sensitivity to the scenarios in tasks. We proposed using the many-facet Rasch measurement (MFRM) to examine the CIV because this measurement model can compare different CIV factors on a shared scale. To demonstrate how to apply this framework, we applied the framework to a video-based science teacher pedagogical content knowledge (PCK) assessment, including two tasks, each with three scenarios. Results for task I, which assessed teachers’ analysis of student thinking , indicate that the CIV due to the variability of the scenarios was substantial, while the CIV due to judging severity and rater scoring sensitivity of the scenarios in teacher responses was not. For task II, which assessed teachers’ analysis of responsive teaching , results showed that the CIV due to the three proposed factors was all substantial. We discuss the conceptual and methodological contributions, and how the results inform item development
Conceptual framework · Construct validity · Developmental psychology · Item response theory · Psychometrics · Rasch model · Statistics · Task Analysis · Variance components · Computer Science · Engineering · Mathematics · Psychology · Science Education and Pedagogy · Social Psychology · Student Assessment and Feedback · Teacher Education and Leadership Studies
Educational measurement.
Statistical Theories of Mental Test Scores
The impact of physics teachers’ pedagogical content knowledge and motivation on students’ achievement and interest
Validating a partial-credit scoring approach for multiple-choice science items
Evaluation of construct-irrelevant variance yielded by machine and human scoring of a science teacher PCK constructed response assessment
Automatic Coding of Short Text Responses via Clustering in Educational Assessment
Generalizability theory
Facilitating the development of preservice teachers' Pedagogical Content Knowledge of literacy and agentic identities
Understanding affordances and challenges of three types of video for teacher professional development
When Do Girls Prefer Football to Fashion? An analysis of female underachievement in relation to ‘realistic’ mathematic contexts
Construct validity in psychological tests
An argument-based approach to validity
Measuring the Involvement Construct
| Unique citing works | 4 |
|---|---|
| Citations per year | 1 |
| Citation span | 2022 - 2026 (5) |
| Citation velocity | current |
| Highly cited | No |
| Citation types | Neutral: 4 |