Using Portfolios in Program Evaluation

An Investigation of Interrater Reliability

Bibliographic Data

ID12470561
AuthorsRobert L Johnson (0000-0002-5938-012X, College of Education, Department of Educational Psychology and Center for Excellence in the Assessment of Student Learning, University of South Carolina at Columbia, Columbia, SC 29208, corresponding author), Fred McDaniel, Marjorie J Willeke
Year2000
Volume21
Issue1
Pages65-80
Publication date2000-03-01
Peer ReviewedYes
Open AccessYes
TypeARTICLE
VenueAmerican Journal of Evaluation (JOURNAL)
Journal identifiersISSN: 1098-2140 • E-ISSN: 1557-0878
PublisherSAGE Publishing (PUBLISHER • US)
DOI10.1177/109821400002100106
OpenAlexW4230008109
LanguageEN
Citations received2
References cited18

Portfolios and other open-ended assessments are increasingly incorporated into evaluations and testing programs. However, questions about the reliability of such assessments continue to be raised. After reviewing forces that may be leading to increased interest in and use of portfolio assessment, we investigate the interrater reliability of a portfolio assessment used in a small-scale program evaluation. Three types of portfolio scores were investigated—analytic, combined analytic (formed by summing across analytic scores), and holistic. The interrater reliability coefficient was highest for summed analytic scores ( r 5 .86). Results indicate that at least three raters are required to obtain acceptable levels of reliability for holistic and individual analytic scores

Developmental psychology · Economics · Evaluation methods · Inter-rater reliability · Portfolio · Rating scale · Reliability (semiconductor · Reliability engineering · Scale (ratio · Statistics · Educational and Psychological Assessments · Engineering · Mathematics · Psychology · Reflective Practices in Education · Student Assessment and Feedback

Tools

Open DOISci-Hub