Reconsidering Cutoff Points in the General Method of Empirical Q-Matrix Validation

Cognitive diagnosis models (CDMs) are latent class multidimensional statistical models that help classify people accurately by using a set of discrete latent variables, commonly referred to as attributes. These models require a Q-matrix that indicates the attributes involved in each item. A potential problem is that the Q-matrix construction process, typically performed by domain experts, is subjective in nature. This might lead to the existence of Q-matrix misspecifications that can lead to inaccurate classifications. For this reason, several empirical Q-matrix validation methods have been developed in the recent years. de la Torre and Chiu proposed one of the most popular methods, based on a discrimination index. However, some questions related to the usefulness of the method with empirical data remained open due the restricted number of conditions examined, and the use of a unique cutoff point ( EPS) regardless of the data conditions. This article includes two simulation studies to test this validation method under a wider range of conditions, with the purpose of providing it with a higher generalization, and to empirically determine the most suitable EPS considering the data conditions. Results show a good overall performance of the method, the relevance of the different studied factors, and that using a single indiscriminate EPS is not acceptable. Specific guidelines for selecting an appropriate EPS are provided in the discussion

Cutoff · Data mining · Econometrics · Empirical research · Generalization · Item response theory · Latent class model · Latent variable · Latent variable model · Machine learning · Matrix (chemical analysis) · Psychometrics · Range (aeronautics) · Relevance (law) · Set (abstract data type) · Statistics · Advanced Statistical Methods and Models · Artificial Intelligence · Computer Science · Mathematics · Multi-Criteria Decision Making · Psychometric Methodologies and Testing

• An empirical Q-matrix validation method using complete information matrix in cognitive diagnostic models

  Yanlou Liu, Qiongqiong WU•Acta Psychologica Sinica•2023

  Incluir item

• Stata Tip 63

  Open Access•Christopher F Baum•The Stata Journal: Promoting…•2008

  Incluir item

• The Generalized Dina Model Framework

  Open Access•Jimmy de la Torre•Psychometrika•2011

  Incluir item

• Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory

  Open Access•Brian W Junker, Klaas Sijtsma•Applied Psychological Measurement•2001

  Incluir item

• Measurement of psychological disorders using cognitive diagnosis models.

  Jonathan Templin, Jonathan L Templin et al.•Psychological Methods•2006

  Incluir item

• Higher-Order Latent Trait Models for Cognitive Diagnosis

  Open Access•Jimmy de la Torre, Jeffrey A Douglas•Psychometrika•2004

  Incluir item

• The Effects of Q-Matrix Misspecification on Parameter Estimates and Classification Accuracy in the Dina Model

  Open Access•André A Rupp, André Rupp et al.•Educational and Psychological…•2008

  Incluir item

• The Effects of Q-Matrix Design on Classification Accuracy in the Log-Linear Cognitive Diagnosis Model

  Open Access•Matthew J Madison, Laine Bradshaw et al.•Educational and Psychological…•2015

  Incluir item

• The Impact of Q-Matrix Designs on Diagnostic Classification Accuracy in the Presence of Attribute Hierarchies

  Open Access•Ren Liu, Anne Corinne Huggins-Manley et al.•Educational and Psychological…•2017

  Incluir item

• Analysis of Clinical Data From Cognitive Diagnosis Modeling Framework

  Jimmy de la Torre, L Andries Van Der Ark et al.•Measurement and Evaluation in…•2015

  Incluir item