The Expertise Effect on Web Accessibility Evaluation Methods
Bibliographic Data
| ID | 15216267 |
|---|---|
| Authors | Giorgio Brajnik (0000-0002-0634-7676, Università di Udine, corresponding author), Yeliz Yeşilada (0000-0001-8501-0205, Middle East Technical University), Simon Harper (0000-0001-9301-5049, University of Manchester) |
| Year | 2011 |
| Volume | 26 |
| Issue | 3 |
| Pages | 246-283 |
| Publication date | 2011-08-30 |
| Peer Reviewed | Yes |
| Open Access | Yes |
| Type | ARTICLE |
| Venue | Human-Computer Interaction (JOURNAL) |
| Journal identifiers | ISSN: 0737-0024 • E-ISSN: 1532-7051 |
| Publisher | Taylor & Francis (PUBLISHER • GB) |
| DOI | 10.1080/07370024.2011.601670 |
| OpenAlex | W1535223020 |
| Language | EN |
| Citations received | 4 |
| References cited | 12 |
Web accessibility means that disabled people can effectively perceive, understand, navigate, and interact with the web. Web accessibility evaluation methods are needed to validate the accessibility of web pages. However, the role of subjectivity and of expertise in such methods is unknown and has not previously been studied. This article investigates the effect of expertise in web accessibility evaluation methods by conducting a Barrier Walkthrough (BW) study with 19 expert and 57 nonexpert judges. The BW method is an evaluation method that can be used to manually assess the accessibility of web pages for different user groups such as motor impaired, low vision, blind, and mobile users.\n\nOur results show that expertise matters, and even though the effect of expertise varies depending on the metric used to measure quality, the level of expertise is an important factor in the quality of accessibility evaluation of web pages. In brief, when pages are evaluated with nonexperts, we observe a drop in validity and reliability. We also observe a negative monotonic relationship between number of judges and reproducibility: more evaluators mean more diverse outputs. After five experts, reproducibility stabilizes, but this is not the case with nonexperts. The ability to detect all the problems increases with the number of judges: With 3 experts all problems can be found, but for such a level 14 nonexperts are needed. Even though our data show that experts rated pages differently, the difference is quite small. Finally, compared to nonexperts, experts spent much less time and the variability among them is smaller, they were significantly more confident, and they rated themselves as being more productive. The article discusses practical implications regarding how BW results should be interpreted, how to recruit evaluators, and what happens when more than one evaluator is hired.\n\nSupplemental materials are available for this article. Go to the publisher's online edition of Human–Computer Interaction for statistical details and additional measures for this article
Quality (philosophy · Web accessibility · Web Accessibility Initiative · Web analytics · Web application security · Web design · Web development · Web navigation · Web page · Web standards · World Wide Web · Assistive Technology in Communication and Mobility · Computer Science · Digital Accessibility for Disabilities · Subtitles and Audiovisual Media
You can't be what you can't read
A comparative analysis of single-page and site-wide tools with the Optimal-EM framework
How compliance with web accessibility standards shapes the experiences of users with and without disabilities
Reflections of a community-based participatory researcher from the intersection of disability advocacy, engineering, and the academy
| Unique citing works | 4 |
|---|---|
| Citations per year | 0.4 |
| Citation span | 2016 - 2026 (11) |
| Citation velocity | current |
| Highly cited | No |
| Citation types | Neutral: 3 |