Differential Item Functioning Effect Size Use for Validity Information

Bibliographic Data

ID20284166
AuthorsW Holmes Finch (0000-0003-0393-2906, Ball State University, Muncie, IN, USA, corresponding author), María Dolores Hidalgo Montesinos (0000-0002-6256-718X, Universidad de Murcia), Brian F French (0000-0002-3896-7888, Washington State University, Pullman, USA), Maria Hernandez Finch (Ball State University, Muncie, IN, USA)
Year2025
Volume85
Issue2
Pages258-276
Publication date2025-04-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueEducational and Psychological Measurement (JOURNAL)
Journal identifiersISSN: 0013-1644 • E-ISSN: 1552-3888
PublisherSAGE Publications (PUBLISHER • US)
DOI10.1177/00131644241293694
PMID39583008
OpenAlexW4404618206
LanguageEN
References cited24

There has been an emphasis on effect sizes for differential item functioning (DIF) with the purpose to understand the magnitude of the differences that are detected through statistical significance testing. Several different effect sizes have been suggested that correspond to the method used for analysis, as have different guidelines for interpretation. The purpose of this simulation study was to compare the performance of the DIF effect size measures described for quantifying and comparing the amount of DIF in two assessments. Several factors were manipulated that were thought to influence the effect sizes or are known to influence DIF detection. This study asked the following two questions. First, do the effect sizes accurately capture aggregate DIF across items? Second, do effect sizes accurately identify which assessment has the least amount of DIF? We highlight effect sizes that had support for performing well across several simulated conditions. We also apply these effect sizes to a real data set to provide an example. Results of the study revealed that the log odds ratio of fixed effects (Ln OR ̄ FE ) and the variance of the Mantel–Haenszel log odds ratio ( τ ^ 2 ) were most accurate for identifying which test contains more DIF. We point to future directions with this work to aid the continued focus on effect sizes to understand DIF magnitude

Differential effects · Differential item functioning · Econometrics · Item response theory · Logistic regression · Odds · Psychometrics · Sample size determination · Set (abstract data type) · Statistical hypothesis testing · Statistics · Variance (accounting) · Cognitive Abilities and Testing · Computer Science · Mathematics · Motivation and Self-Concept in Sports · Psychology · Psychometric Methodologies and Testing

Tools

Open DOIOpen Access