Accessibility Evaluators Aren’t Worthless
As seen on our Resources page, there are a number of web accessibility evaluation tools available to developers. Most accessible web developers fall into three groups as it pertains to the value of these tools. Either they love them, hate them, or don’t really understand how to use them. The purpose of this article is to encourage those who love them to not overly depend on them, those who hate them to stop being haters, and those who don’t understand them to come away with an open mind.
The problem with these tools, the reason some revile them, is that they can lead to a false sense of security among those users who don’t really understand their limitations. It stands to reason. Developers see the “Passed” status for a “AAA” level of compliance, and the next thing you know they’re going public with that information. And as a result of this innocent action, others are led to believe a site reaches a particular level of accessibility when in fact it doesn’t. This exacerbates the problem. This isn’t, however, the fault of the evaluation tool. They do have a legitimate value.
Where’s the Value?
I build my sites and, before going live with them, I always run them through an automated accessibility evaluation tool, at least on the template level. I used to use Bobby or Cynthia for this, but Bobby’s now gone, so now I use only Cynthia (which some people dislike even more than they hated Bobby). To me it doesn’t matter which one I use because for what I use them for it has little bearing. For my needs, Cynthia is just as effective and reliable as Bobby ever was. The key here is “my needs.” I don’t expect from these handy free tools the level of sophistication one might get from a top-shelf evaluator like those offered by Deque Systems, but I don’t expect what they can’t deliver. I instead focus on what they can do for me.
I find value in just about any error detected. If it’s caught by Cynthia, then something is probably wrong. One possible trigger I personally find exceptionally valuable is the presence of link text used on links pointing to multiple locations (Web Content Accessibility Guidelines checkpoint 13.1, better defined under the techniques). This can be incredibly difficult to spot on one’s own (for me, anyway). Using a tool in this capacity is a real time-saver. There are many uses for these tools, and some things to watch out for, as described in the Guild of Accessible Web Designers (GAWDS) article by Grant Broome titled Automated testing - How useful is it?.
Obviously you’ll have to explore and find value where it exists for you, but to completely dismiss these these tools because of their shortcomings isn’t fair. Just remember that if it says your site passes, it isn’t necessarily so — go through the entire checklist confirming manually what the tool suggests you confirm. Once that’s done you may claim that your site is accessible, but not before. These tools are just that: tools, and tools are something to help you get something done. You can beat on a can with a can opener all day long, probably denting it into some form of submission, but the can will remain sealed. It’s not the can opener’s fault the can remained sealed, the tool just wasn’t properly used.
What value, if any, do you place on automated accessibility tools? How do these tools make your life easier? How do they bail you out?
Craig responds:
Posted: May 6th, 2008 at 9:34 am →
I use accessibility tools such as Cythia, but only to double-check that I’ve not made fundamental coding errors. After all, it’s possible to make a site that passes evaluation, but couldn’t really be considered accessible.
It is possible that unscrupulous developers can use these tools to put a stamp of authority on a shoddy website. However, it will still be promoting accessibility as “a good thing” and that developer will be encouraged to update their skills.
Mel Pedley responds:
Posted: May 6th, 2008 at 2:41 pm →
I’m not so sure. A stamp of authority only has real value when it can be trusted. If people persist in using these badges on sites that patently don’t achieve the levels of compliance stated, it is the badge/level that becomes ultimately devalued. Developers have to be wary of just ticking boxes and satisfying dumb accessibility parsing tools.
Don’t get me wrong. like yourself and Mike, I find these tools are really helpful in highlighting potential issues or errors but I reserve the right to disagree with them, or even ignore them completely in some circumstances. The final arbiters have to be the appropriate guidelines and the users themselves.
The tools are simply a means to an end. Not The End.
Mike Cherim responds:
Posted: May 6th, 2008 at 3:01 pm →
I’ve ignored them before too, knowing something wasn’t an issue even though the validator did. Usually though I can find the cause which is sometimes valuable as well.
John Faulds responds:
Posted: May 6th, 2008 at 5:45 pm →
It seems one of the tools I use most often is missing from your list Mike: Total Validator. I don’t use Cynthia but I’m guessing that Total Validator picks up the same sorts of mistakes that Cynthia does. I also find Gez Lemon’s Colour Contrast Analyser extension for Firefox quite useful for pointing out when I haven’t specified background-colors when using background-images. Another I use quite often is IBM’s aDesigner which also seems to be missing from your list of resources.
Mike Cherim responds:
Posted: May 6th, 2008 at 10:20 pm →
@John: Gez’s analyzer’s on our list, at least the site version. I have the add-on for Firefox too. It’s awesome. As far as the other, I’ve heard of neither, and both are cool. The IBM tool aDesigner looks like some very helpful software what with its simulations (no blind empathy there), and Total Validator… the name says it all, even spelling. I like it. Thank you very much for sharing them. They have been added to our ever-growing list.
Dave Woods responds:
Posted: May 7th, 2008 at 2:45 am →
Hi Mike, I completely agree and I follow exactly the same approach as you in that Cynthia is great as a first step to check that there are no glaringly obvious problems with a site. It’s also obviously important that web developers understand that if they want their site to be accessible then they should have an in-depth understanding of the WAI Guidelines and not simply rely on tools like Cynthia as a be all and end all solution.
It’s also probably worth mentioning that another useful “tool” towards accessibility validation is disabling images, CSS JavaScript independently of each other as well as any combination of the three. It’s not a tool so to speak but is another free and easy way to help spot any accessibility issues.
Dave Woods - HTML, CSS, Web Design » Useful Web Development Links (7th May 2008) responds:
Posted: May 7th, 2008 at 3:02 am →
[…] Accessibility Evaluators Aren’t Worthless […]
David Zemens - 1955 Design responds:
Posted: May 7th, 2008 at 8:41 am →
I certainly understand the need to manually investigate each checklist warning to assure full compliance. No question about it.
As a fledgling web developer who is striving to do things the right way, a tool such as Cynthia gives me confidence that I am on the right track. I confess to not manually checking each warning. For the moment, I am just happy that I can generally achieve compliance with the automated validation. I suppose this is the first step to reaching a higher level, but I still have a long way to go.
Great article and great comments!
Tim Harshbarger responds:
Posted: May 10th, 2008 at 12:36 pm →
I know that there is a concern about developers misusing evaluation tools to certify the accesibility of a site. However, I think there is an assumption that accessibility claims were or are important to people using a site. I don’t think users have ever paid that much attention to accessibility claims. I expect most of them have little or no familiarity with accessibility guidelines and standards. I expect most people judge a site’s commitment to accessibility through personal experience.
The tools can add value in ensuring that people do have a positive accessible experience with a site, but I am uncertain misuse of the tools to make outrageous claims of accessibility negatively impact users that much.
Simosito responds:
Posted: May 12th, 2008 at 8:26 am →
I use F.A.E () and it is different from the others.
Maybe it is the future of validators.
Nic responds:
Posted: May 25th, 2008 at 12:43 am →
Posting a response a bit late, I hadn’t been on your site for yonks! It’s funny, I wrote an article about Automated Testing Tools last month!
I use them, no worries. It’s a tool. It’s a starting point. They don’t always get it right. But usualy.
As for badges… The proof is in the pudding, as they say, badge or no badge, it’s user experience that really matters.
dani responds:
Posted: May 27th, 2008 at 1:04 pm →
I choose ATRC web checker for web accessibility and validome.org for X/HTML validator