Assessing the accuracy and consistency of language proficiency classification under competing measurement models
Bibliographic Data
| ID | 19380992 |
|---|---|
| Authors | Bo Zhang (0000-0003-0986-0018, University of Wisconsin–Milwaukee, corresponding author), Bo Zhang Bo Zhang (University of Wisconsin–Milwaukee) |
| Year | 2010 |
| Volume | 27 |
| Issue | 1 |
| Pages | 119-140 |
| Publication date | 2010-01-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Language Testing (JOURNAL) |
| Journal identifiers | ISSN: 0265-5322 • E-ISSN: 1477-0946 |
| Publisher | SAGE Publications (PUBLISHER • US) |
| DOI | 10.1177/0265532209347363 |
| OpenAlex | W2161431965 |
| Language | EN |
| Citations received | 10 |
| References cited | 29 |
This article investigates how measurement models and statistical procedures can be applied to estimate the accuracy of proficiency classification in language testing. The paper starts with a concise introduction of four measurement models: the classical test theory (CTT) model, the dichotomous item response theory (IRT) model, the testlet response theory (TRT) model, and the polytomous item response theory (Poly-IRT) model. Following this, two classification procedures are presented: the Livingston and Lewis method for CTT and the Rudner method for the three IRT-based models. The utility of these models and procedures are then evaluated by examining the accuracy of classifying 5000 language test takers from a large-scale language certification examination into two proficiency categories. The most important finding is that the testlet format (multiple questions based on one prompt), which language tests usually rely on, has a great impact on the proficiency classification. All testlets in this study show a strong testlet effect. Hence, the TRT model is recommended for proficiency classification. Using the standard IRT model would inflate the classification accuracy due to the underestimated measurement error. Meanwhile, using the Poly-IRT model would give slightly less accurate classification results. Concerning the CTT model, while its classification accuracy is comparable to that of the TRT, there exists considerable inconsistency between their classification results
Classical test theory · Econometrics · Item response theory · Language assessment · Language proficiency · Machine learning · Mathematics education · Natural language processing · Polytomous Rasch model · Psychometrics · Statistics · Computer Science · Educational Technology and Assessment · Mathematics · Psychology · Psychometric Methodologies and Testing · Student Assessment and Feedback · Artificial Intelligence
La méthode de notation d’un questionnaire importe-t-elle
A systematic review of item response theory in language assessment
Applying unidimensional and multidimensional item response theory models in testlet-based reading assessment
IRT-based classification analysis of an English language reading proficiency subtest
Examining testlet effects in the TestDaF listening section
Developing individualized feedback for listening assessment
Do decision rules matter? A descriptive study of English language proficiency assessment classifications for English-language learners and native English speakers in fifth grade
The Utility of the Bifactor Method for Unidimensionality Assessment When Other Methods Disagree
The economics of Japanese
Assessing Testlet Effect, Impact, Differential Testlet, and Item Functioning Using Cross-Classified Multilevel Measurement Modeling
Item Response Theory
Scaling Performance Assessments
Theoretical Bases of Communicative Approaches to Second Language Teaching and Testing
A latent trait method for measuring a dimension in second language proficiency
Item Response Theory and the validation of an ESP test for health professionals
Person dimensionality in language test validation
The Trait Structure of Cloze Test Scores
What Does Language Testing Have to Offer
The Cloze Procedure and Proficiency in English as a Foreign Language
Coefficient alpha and the internal structure of tests
Theoretical Bases of Communicative Approaches to Second Language Teaching and Testing
Cloze Tests of Second Language Proficiency and What They Measure1
| Unique citing works | 10 |
|---|---|
| Citations per year | 0.83 |
| Citation span | 2014 - 2024 (11) |
| Citation velocity | recent |
| Highly cited | No |
| Citation types | Neutral: 10 |