A Study of the Evaluator Effect in Usability Testing
Bibliographic Data
| ID | 15216155 |
|---|---|
| Authors | Kasper Hornbæk (0000-0001-8742-1198, University of Copenhagen, corresponding author), Erik Frøkjær (0000-0002-6627-9396, University of Copenhagen) |
| Year | 2008 |
| Volume | 23 |
| Issue | 3 |
| Pages | 251-277 |
| Publication date | 2008-08-29 |
| Open Access | No |
| Type | ARTICLE |
| Venue | Human-Computer Interaction (JOURNAL) |
| Journal identifiers | ISSN: 0737-0024 • E-ISSN: 1532-7051 |
| Publisher | Taylor & Francis (PUBLISHER • GB) |
| DOI | 10.1080/07370020802278205 |
| OpenAlex | W1974725562 |
| Language | EN |
| Citations received | 1 |
| References cited | 8 |
The evaluator effect names the observation that usability evaluators in similar conditions identify substantially different sets of usability problems. Yet little is known about the factors involved in the evaluator effect. We present a study of 50 novice evaluators' usability tests and subsequent comparisons, in teams and individually, of the resulting usability problems. The same problems were analyzed independently by 10 human–computer interaction experts. The study shows an agreement between evaluators of about 40%, indicating a substantial evaluator effect. Team matching of problems following the individual matching appears to improve the agreement, and evaluators express greater satisfaction with the teams' matchings. The matchings of individuals, teams, and independent experts show evaluator effects of similar sizes; yet individuals, teams, and independent experts fundamentally disagree about which problems are similar. Previous claims in the literature about the evaluator effect are challenged by the large variability in the matching of usability problems; we identify matching as a key determinant of the evaluator effect. An alternative view of usability problems and evaluator agreement is proposed in which matching is seen as an activity that helps to make sense of usability problems and where the existence of a correct matching is not assumed
Human–computer interaction · Matching (statistics · Statistics · Usability · Usability engineering · Usability inspection · Web usability · Computer Science · Mathematics · Software Engineering Techniques and Practices · Team Dynamics and Performance · Usability and User Interface Design
| Citations per year | 0.07 |
|---|---|
| Citation span | 2011 - 2011 (1) |
| Highly cited | No |