Setting Standards With Multiple-Choice Tests
A Preliminary Intended-User Evaluation of SmartStandardSet
Bibliographic Data
| ID | 22163839 |
|---|---|
| Authors | Gavin T L Brown (0000-0002-8352-2351, University of Auckland, corresponding author), Paul Denny (0000-0002-5150-9806, University of Auckland), David L San Jose (University of Auckland), Ellen Li (0000-0002-1141-0406, University of Auckland) |
| Year | 2021 |
| Volume | 6 |
| Publication date | 2021-08-31 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Frontiers in Education (JOURNAL) |
| Journal identifiers | ISSN: 2504-284X • E-ISSN: 2504-284X |
| Publisher | Frontiers Media SA (PUBLISHER • CH) |
| DOI | 10.3389/feduc.2021.735088 |
| OpenAlex | W3198870776 |
| Language | EN |
| References cited | 29 |
Software that easily helps higher education instructors to remove poor quality items and set appropriate grade boundaries is generally lacking. To address these challenges, the SmartStandardSet system provides a graphical-user interface for removing defective items, weighting student scores using a two-parameter model IRT score analysis, and a mechanism for standard-setting. We evaluated the system through a series of six interviews with teachers and six focus groups involving 19 students to understand how key stakeholders would view the use of the tool in practice. Generally, both groups of participants reported high levels of feasibility, accuracy, and utility in SmartStandardSet’s statistical scoring of items and score calculation for test-takers. Teachers indicated the data displays would help them improve future test items; students indicated the system would be fairer and would motivate greater effort on more difficult test items. However, both groups had concerns about implementing the system without institutional policy endorsement. Students specifically were concerned that academics may set grade boundaries on arbitrary and invalid grounds. Our results provide useful insights into the perceived benefits of using the tool for standard setting, and suggest concrete next steps for gaining wider acceptance that will be the focus of future work
Focus group · Graphical user interface · Knowledge management · Multiple choice · Significant difference · Statistics · Weighting · Computer Science · Engineering · Innovative Teaching Methods · Mathematics · Online Learning and Analytics · Student Assessment and Feedback · Software
Educational measurement.
The technology acceptance model (TAM)
Developing evaluative judgement
A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment
Mirt
The theory of planned behavior
How sure can we be that a student really failed? On the measurement precision of individual pass-fail decisions from the perspective of Item Response Theory
Knowledge, application and how about competence? Qualitative assessment of multiple-choice questions for dental students
A Practical Iterative Framework for Qualitative Data Analysis
| Citation velocity | historical |
|---|---|
| Highly cited | No |