Determining the scoring validity of a co-constructed Cefr-based rating scale
Bibliographic Data
| ID | 19381027 |
|---|---|
| Authors | Bart Deygers (0000-0002-3198-0449, KU Leuven), Koen Van Gorp (0000-0002-2033-3852, KU Leuven) |
| Year | 2015 |
| Volume | 32 |
| Issue | 4 |
| Pages | 521-541 |
| Publication date | 2015-10-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Language Testing (JOURNAL) |
| Journal identifiers | ISSN: 0265-5322 • E-ISSN: 1477-0946 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/0265532215575626 |
| OpenAlex | W2327696282 |
| Language | EN |
| Citations received | 7 |
| References cited | 31 |
Considering scoring validity as encompassing both reliable rating scale use and valid descriptor interpretation, this study reports on the validation of a CEFR-based scale that was co-constructed and used by novice raters. The research questions this paper wishes to answer are (a) whether it is possible to construct a CEFR-based rating scale with novice raters that yields reliable ratings and (b) allows for a uniform interpretation of the descriptors. Additionally, this study focuses on the question whether co-constructing a rating scale with novice raters helps to stimulate a shared interpretation of the descriptors over time. For this study, six novice raters employed a CEFR-based scale that had been co-constructed by themselves and 14 peers to rate 200 spoken and written performances in a missing data design. The quantitative data were analysed using item response theory, classical test theory and principal component analysis. The focus group data, collected after the rating process, were transcribed and coded using both a priori and inductive coding. The results indicate that novice raters can reliably use the CEFR-based rating scale, but that the interpretations of the descriptors, in spite of training and co-construction, are not as homogeneous as the inter-rater reliability would suggest
Construct validity · Developmental psychology · Natural language processing · Psychometrics · Rating scale · Computer Science · Educational and Psychological Assessments · Psychology · Psychometric Methodologies and Testing · Student Assessment and Feedback · Artificial Intelligence
Rating Scales and Rubrics in Language Assessment
Learner agency and assessment co-adaptation from a Complex Dynamic Systems Theory perspective
Developing a rating scale for written intralinguistic mediation in a local context
Begrijpelijkheid en accent bij beginnende NT2-leerders
The longitudinal stability of rating characteristics in an EFL examination
Revisiting rating scale development for rater-mediated language performance assessments
Validation of rating processes within an argument-based framework
Educational measurement.
Assessing Writing
Language Testing and Validation
Qualitative Data Analysis
Language Proficiency in Native and Nonnative Speakers
Rater Experience, Rating Scale Length, and Judgments of L2 Pronunciation
The Kappa Statistic in Reliability Studies
Effective rating scale development for speaking tests
Diagnostic assessment of writing
The Rasch wars
Explaining ESL essay holistic scores
Limitations of the Common European Framework for developing comparable examinations and tests
Rater types in writing performance assessments
Relationship of Toefl iBT® scores to academic performance
Second Language Fluency
Where to begin? Grappling with how to use participant interaction in focus group design
| Citations per year | 0.88 |
|---|---|
| Citation span | 2018 - 2026 (9) |
| Highly cited | No |