Empowering Expert Judgment

A Data-Driven Decision Framework for Standard Setting in High-Dimensional and Data-Scarce Assessments

Bibliographic Data

ID20650102
AuthorsTianpeng Zheng (0000-0002-3594-9078, Peking University), Zhehan Jiang (0000-0002-1376-9439, Peking University, corresponding author), Z Y Guo (0000-0002-9522-620X, Peking University), Yuanfang Liu (0000-0002-5058-7685, Peking University)
Year2026
Pages00131644251405406-00131644251405406
Publication date2026-01-02
Open AccessYes
TypeARTICLE
VenueEducational and Psychological Measurement (JOURNAL)
Journal identifiersISSN: 0013-1644 • E-ISSN: 1552-3888
PublisherSAGE Publications (PUBLISHER • US)
DOI10.1177/00131644251405406
PMID41492416
OpenAlexW7118093825
LanguageEN
References cited43

A critical methodological challenge in standard setting arises in small-sample, high-dimensional contexts where the number of items substantially exceeds the number of examinees. Under such conditions, conventional data-driven methods that rely on parametric models (e.g., item response theory) often become unstable or fail due to unreliable parameter estimation. This study investigates two families of data-driven methods: information-theoretic and unsupervised clustering, offering a potential solution to this challenge. Using a Monte Carlo simulation, we systematically evaluate 15 such methods to establish an evidence-based framework for practice. The simulation manipulated five factors, including sample size, the item-to-examinee ratio, mixture proportions, item quality, and ability separation. Method performance was evaluated using multiple criteria, including Relative Error, Classification Accuracy, Sensitivity, Specificity, and Youden’s Index. Results indicated that no single method is universally superior; the optimal choice depends on the examinee mixture proportion. Specifically, the information-theoretic method QIR (quantile information ratio) excelled in scenarios with a dominant non-competent group, where high specificity was critical. Conversely, in highly selective contexts with balanced proficiency groups, the clustering methods CHI (Calinski-Harabasz index) and sum of squared error (SSE) demonstrated the highest classification effectiveness. Bayesian kernel density estimation (BKDE), however, consistently performed as a robust, balanced method across conditions. These findings provide practitioners with a clear decision framework for selecting a defensible, data-driven standard-setting method when traditional approaches are infeasible

Bayesian probability · Cluster analysis · Decision theory · Item response theory · Kernel density estimation · Monte Carlo method · Parametric statistics · Sample (material) · Psychometric Methodologies and Testing · Reliability and Agreement in Measurement · Statistical Methods and Bayesian Inference

Tools

Open DOIOpen Access