Paper Info
Title
The Expertise Effect on Web Accessibility Evaluation Methods.
Abstract
Web accessibility means that disabled people can effectively perceive, understand, navigate, and interact with the web. Web accessibility evaluation methods are needed to validate the accessibility of web pages. However, the role of subjectivity and of expertise in such methods is unknown and has not previously been studied. This article investigates the effect of expertise in web accessibility evaluation methods by conducting a Barrier Walkthrough (BW) study with 19 expert and 57 nonexpert judges. The BW method is an evaluation method that can be used to manually assess the accessibility of web pages for different user groups such as motor impaired, low vision, blind, and mobile users. Our results show that expertise matters, and even though the effect of expertise varies depending on the metric used to measure quality, the level of expertise is an important factor in the quality of accessibility evaluation of web pages. In brief, when pages are evaluated with nonexperts, we observe a drop in validity and reliability. We also observe a negative monotonic relationship between number of judges and reproducibility: more evaluators mean more diverse outputs. After five experts, reproducibility stabilizes, but this is not the case with nonexperts. The ability to detect all the problems increases with the number of judges: With 3 experts all problems can be found, but for such a level 14 nonexperts are needed. Even though our data show that experts rated pages differently, the difference is quite small. Finally, compared to nonexperts, experts spent much less time and the variability among them is smaller, they were significantly more confident, and they rated themselves as being more productive. The article discusses practical implications regarding how BW results should be interpreted, how to recruit evaluators, and what happens when more than one evaluator is hired. Supplemental materials are available for this article. Go to the publisher's online edition of Human-Computer Interaction for statistical details and additional measures for this article.
Year
DOI
Venue
2011
10.1080/07370024.2011.601670
HUMAN-COMPUTER INTERACTION
Keywords
Field
DocType
human computer interaction,web accessibility,accessibility,web pages,mobile web
Web development,Web design,Web Accessibility Initiative,World Wide Web,Web page,Computer science,Web standards,Web modeling,Human–computer interaction,Web navigation,Web accessibility
Journal
Volume
Issue
ISSN
26.0
3
0737-0024
Citations 
PageRank 
References 
12
0.74
21
Authors
3
Name
Order
Citations
PageRank
Giorgio Brajnik164562.79
yeliz yesilada256674.67
Simon Harper31105140.48