The Difficulty with Setting a Standard Based on Difficulty
The Role of Validity in Determining Assessment Standards
Bibliographic Data
| ID | 22047902 |
|---|---|
| Authors | Steven Ashley Burr (University of Plymouth), Daniel Zahra (0000-0001-6534-6916, Peninsula College of Medicine and Dentistry), Iain M Robinson (0000-0003-3770-9710, Lancaster University) |
| Year | 2026 |
| Volume | 16 |
| Issue | 3 |
| Pages | 488 |
| Publication date | 2026-03-21 |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Education Sciences (JOURNAL) |
| Journal identifiers | ISSN: 2227-7102 • E-ISSN: 2227-7102 |
| Publisher | MDPI AG (PUBLISHER • IT) |
| DOI | 10.3390/educsci16030488 |
| OpenAlex | W7140127294 |
| Language | EN |
| References cited | 49 |
When setting standards for high-stakes assessments, it is necessary to be clear whether the primary criterion should be importance or difficulty. While standard setting practices frequently rely on the difficulty to determine passing or failing thresholds, this can undermine the intended inferences and uses of assessments. Standard setting based on difficulty conflates importance with subjective perceptions of difficulty, misguiding decisions about what candidates need to demonstrate for safe and effective practice. We explore several reasons why difficulty does not align with the aims of assessment and how prioritising importance can better reflect the intended uses and inferences of a test. Our analysis incorporates: (1) the relevance of validity to setting standards, (2) potential misunderstandings of norm- and criterion referencing, (3) critical differentiation between importance and difficulty, (4) reasons for prioritising validity, (5) the variability of candidates’ difficulty perceptions, (6) challenges of assessor judgments, (7) the need for clear definitions of competence, and (8) the appropriate use of ‘would’ vs. ‘should’ in establishing standards. Ultimately, a valid standard reflects the ability to demonstrate performance of what is important, not difficult, in the workplace
Evaluation methods · External validity · Inference · Perception · Competency Development and Evaluation · Psychometric Methodologies and Testing · Student Assessment and Feedback
Human Cognitive Abilities
Monitoring one's own knowledge during study
Validity
Cognitive Load During Problem Solving
“But is it fair?”
A Preliminary Investigation of Two Procedures for Setting Examination Standards
The Improvement of Estimates of Test Difficulty
Should Taxonomic Levels be Considered in Developing Examinations
Absolute Grading Standards for Objective Tests
The Prediction of Absolute Item Difficulty By Ranking and Estimating Techniques
A Meta-Analysis of the Effects of Modifications to the Angoff Method on Cutoff Scores and Judgment Consensus
The role of perceptions of task difficulty in relation to self-perceptions of ability, intrinsic value, attainment value, and performance
Instructional technology and the measurement of learing outcomes
Validity of psychological assessment
Theory of fluid and crystallized intelligence
| Highly cited | No |
|---|