Assessing the Psychometric Properties of Quality Experience in Undergraduate Research Using Item Response Theory
Bibliographic Data
| ID | 21752011 |
|---|---|
| Authors | Tien-Ling Hu (0000-0002-6429-6854, William & Mary, corresponding author), Dubravka Svetina (0000-0003-3286-3540, Indiana University Bloomington) |
| Year | 2024 |
| Volume | 65 |
| Issue | 8 |
| Pages | 1965-1991 |
| Publication date | 2024-12-01 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Research in Higher Education (JOURNAL) |
| Journal identifiers | ISSN: 1573-188X • E-ISSN: 0361-0365 |
| Publisher | Springer Science and Business Media LLC (PUBLISHER) |
| DOI | 10.1007/s11162-024-09814-6 |
| OpenAlex | W4402421235 |
| Language | EN |
| References cited | 41 |
Undergraduate research, recognized as one of the High-Impact Practices (HIPs), has demonstrated a positive association with diverse student learning outcomes. Understanding the pivotal quality factors essential for its efficacy is important for enhancing student success. This study evaluates the psychometric properties of survey items employed to gauge the quality of undergraduate research, including alignment with Kuh and O’Donnell’s (2013) eight HIP characteristics, alongside assessments of reliability, validity, and generalizability across demographic groups. The study assesses the validity and reliability of these measures at both the scale and item levels using data from the National Survey of Student Engagement’s (NSSE) HIP Quality Topical Module. The methodological approaches employed include Exploratory Factor Analysis, Parallel Analysis, Item Response Theory, and Differential Item Functioning (DIF). Our findings uncover a misalignment between NSSE’s HIP Quality module items and HIP characteristics, leading to the identification of seven subscales instead of eight. Nevertheless, four subscales—Reflective and Integrative Learning, Real-World Applications, Interactions with Others, and High-Performance Expectations—emerge as valid indicators of undergraduate research experiences. While specific items yield valuable insights at the item level, refinement is recommended for others. Despite the identification of two items exhibiting DIF, their negligible effect sizes suggest that major revisions are unwarranted solely on DIF grounds. This study offers recommendations for item refinement, including the incorporation of new items, wording updates, and tailored utilization of assessment tools within educational institutions. These recommendations are intended to empower educators and researchers to effectively capture the quality dimensions of students’ undergraduate research experiences, thereby fostering their academic success
Economics · Epistemology · Higher education · Item response theory · Mathematics education · Psychometrics · Applied Psychology · Career Development and Diversity · Clinical Psychology · Engineering Education and Curriculum Development · Higher Education Research Studies · Psychology · Social Psychology
An Easy Guide to Factor Analysis
Item Response Theory
Benefits of Undergraduate Research Experiences
Undergraduate Research Experiences Support Science Career Decisions and Active Learning
The national survey of student engagement
Further Investigation of the Performance of S - X2
Survey of Undergraduate Research Experiences (Sure)
Dimensionality assessment of ordered polytomous items with parallel analysis.
Assessment of Course-Based Undergraduate Research Experiences
Mirt
Estimation of Latent Ability Using a Response Pattern of Graded Scores
Minimum Sample Size Recommendations for Conducting Factor Analyses
The link between high-impact practices and student learning
Likelihood-Based Item-Fit Indices for Dichotomous Item Response Theory Models
Cutoff criteria for fit indexes in covariance structure analysis
Factor Analysis and AIC
Estimating the Dimension of a Model
A Second Generation Little Jiffy
Power analysis and determination of sample size for covariance structure modeling.
Evaluation of Parallel Analysis Methods for Determining the Number of Factors
Little Jiffy, Mark Iv
What Experiences Help Students Become Scientists? A Comparative Study of Research and other Sources of Personal and Professional Gains for STEM Undergraduates
Making a Difference in Science Education
Establishing the benefits of research experiences for undergraduates in the sciences
Becoming a scientist
Situated Learning
Alternative Ways of Assessing Model Fit
| Citation velocity | historical |
|---|---|
| Highly cited | No |