A Methodologically Improved Study on Raters’ Personality and Rating Severity in Writing Assessment
Bibliographic Data
| ID | 16915848 |
|---|---|
| Authors | Yu Zhu (0000-0003-1360-0753, Xiamen University), Andy Shui-Lung Fung, Andy S L Fung (Beijing Normal-Hong Kong Baptist University), Liuyan Yang (0000-0002-4211-006X, Xiamen University, corresponding author) |
| Year | 2021 |
| Volume | 11 |
| Issue | 2 |
| Publication date | 2021-04-01 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | SAGE Open (JOURNAL) |
| Journal identifiers | ISSN: 2158-2440 • E-ISSN: 2158-2440 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/21582440211009476 |
| OpenAlex | W3155237168 |
| Language | EN |
| Citations received | 1 |
| References cited | 39 |
Personality is an inherent rater’s characteristic influencing rating severity, but very few studies examined their relationship and the findings were inconclusive. This study aimed to re-investigate the relationship between raters’ personality and rating severity with more control on relevant variables and more reliable analysis of rating severity. Female novice raters ( n = 28) from a demographically homogeneous background were recruited to rate on two occasions essays written by 111 students in an intermediate-level Chinese as a foreign language program. Raters’ personality traits were measured using the complete version of NEO-PI-R. Many-faceted Rasch measurement model and repeated measurement were applied to yield more robust estimates of rating severity. In addition, rating order effect was carefully controlled. Extroversion was found to be positively correlated with severity, r(26) = .495, p = .010. Furthermore, Extroversion was found to be a valid predictor of severity, t(24) = 2.792, p = .010, R 2 = .21, Cohen’s d = .77, Hattie’s r = .37. Practical implications for developing more individualized online rater calibration for large-scale writing assessments were discussed, followed by limitations of the present study
Big Five personality traits · Developmental psychology · Extraversion and introversion · Inter-rater reliability · Personality · Personality Assessment Inventory · Rasch model · Rating scale · Clinical Psychology · Personality Traits and Psychology · Psychology · Psychometric Methodologies and Testing · Social Psychology · Student Assessment and Feedback
A G-Theory Analysis of Rater Effect in ESL Speaking Assessment
The assessment of writing ability
Investigating variability in tasks and rater judgements in a performance test of foreign language speaking
Rater bias patterns in an EFL writing assessment
Using Facets to model rater training effects
Effects of training on raters of ESL compositions
Expertise in evaluating second language compositions
The impact of gender in oral proficiency testing
The Relationship Between the Revised Neo-Personality Inventory and the Myers-Briggs Type Indicator
Exploring gender and oral proficiency interview performance
Do ESL Essay Raters' Evaluation Criteria Change With Experience? A Mixed‐Methods, Cross‐Sectional Study
Performance rating
| Citations per year | 1 |
|---|---|
| Citation span | 2026 - 2026 (1) |
| Highly cited | No |