Communal factors in rater severity and consistency over time in high-stakes oral assessment
Bibliographic Data
| ID | 19380900 |
|---|---|
| Authors | Reeta Neittaanmäki (0000-0001-9741-5584, University of Jyväskylä, corresponding author), Iasonas Lamprianou (0000-0001-7637-615X, University of Cyprus, Cyprus) |
| Year | 2024 |
| Volume | 41 |
| Issue | 3 |
| Pages | 584-605 |
| Publication date | 2024-07-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Language Testing (JOURNAL) |
| Journal identifiers | ISSN: 0265-5322 • E-ISSN: 1477-0946 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/02655322241239363 |
| OpenAlex | W4394684555 |
| Language | EN |
| Citations received | 3 |
| References cited | 37 |
This article focuses on rater severity and consistency and their relation to major changes in the rating system in a high-stakes testing context. The study is based on longitudinal data collected from 2009 to 2019 from the second language (L2) Finnish speaking subtest in the National Certificates of Language Proficiency in Finland. We investigated whether rater severity and consistency changed over that period and whether the changes could be explained by major changes in the rating system, such as the change of lead examiner, the modus of rating and training (on-site or remote), and the composition of the rater group. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analysed using the many-facet Rasch model and generalized linear mixed models. The analyses indicated that raters as a group became somewhat more lenient over time. In addition, the results showed that the rater community and its practices, the lead examiners, and the modus of rating and training can influence rating behaviour. Finally, we elaborate on implications for both research and practice
Developmental psychology · Inter-rater reliability · Internal consistency · Psychometrics · Rating scale · Clinical Psychology · Psychology · Psychometric Methodologies and Testing · Reliability and Agreement in Measurement · Student Assessment and Feedback
Assessing Writing
Fitting Linear Mixed-Effects Models Using lme4
Theoretical Bases of Communicative Approaches to Second Language Teaching and Testing
Test Review
The longitudinal stability of rating characteristics in an EFL examination
Using Facets to model rater training effects
A comparison of newly-trained and experienced raters on a standardized writing assessment
Effects of training on raters of ESL compositions
The development and maintenance of rating quality in performance writing assessment
Investigating the effectiveness of individualized feedback to rating behavior — a longitudinal study
The influence of training and experience on rater performance in scoring spoken language
A many-facet Rasch analysis of the second language group oral discussion task
Rater characteristics and rater bias
Investigation of Rater Effects Using Social Network Analysis and Exponential Random Graph Models
Situated Learning
| Unique citing works | 3 |
|---|---|
| Citations per year | 1.5 |
| Citation span | 2024 - 2026 (3) |
| Citation velocity | current |
| Highly cited | No |
| Citation types | Neutral: 3 |