What we found when we tested tools on the world’s least-accessible webpage

We recently conducted an audit of automated accessibility testing tools. We built a website full of accessibility failures to test them on. We've published our findings here.

In this blog post we talk about what we did and what we discovered.

The pros and cons of automated tools

Automated accessibility testing tools can be used to identify accessibility issues on websites. As the name suggests, they are automated tools that can be run on websites and can identify a number of issues.

There are several available, such as Wave and Tenon. Many of them are free and can be accessed online.

Automated tools can be a useful and cheap way of helping you make a service more accessible. They are quick to run and provide immediate feedback. They can be run across lots of pages. Some can be integrated into the build process, so they can identify issues almost as soon as they are created.

But while it can certainly be helpful to run an automated testing tool on a service, it’s important that teams don’t rely on them too heavily. No tool will be able to pick up every accessibility barrier on a website. So just because a tool hasn’t picked up any accessibility issues on a website, doesn’t mean those issues don’t exist.

And even if they do detect a barrier, sometimes the results they give will be inconclusive or require further investigation. Or even just wrong.

A good analogy is to think of a testing tool as like using a spellchecker. It can certainly help you pick up issues, but it should never be used in isolation. To be most useful, automated tools should be combined with manual inspection and user research.

To help people understand the usefulness – and the limitations – of automated tools, and to help people pick a suitable tool, we carried out an audit of some of the most common tools available.

Choosing the tools to test with

We chose 10 automated testing tools for our audit. We wanted to test the tools that are most commonly used by developers and quality assurance testers. And we wanted to test a large enough number of tools that we would get a variety of results.

We picked all the free tools we were aware of. We also sought suggestions through the cross-government Accessibility Google Group. Here are the tools we tested:

All of these tools are free to use, apart from Sort Site, which has a free trial. Tenon and Wave also have paid versions if you don’t want to run them in your browser.

Testing on the world’s least accessible web page

Once we had decided which tools to work with, we needed a web page to test them on.

We needed a page that was riddled with accessibility problems. One that broke all the accessibility rules. One that featured all kinds of accessibility barriers.

So we built one.

A screenshot of 'the world's least accessible website', which we built to test automated tools on

I worked with Alistair and Richard, my colleagues on the GDS Accessibility team, to create a web-page full of accessibility failures. We refer to it as the world’s least accessible web page.

We filled it with accessibility barriers. At the moment it contains a total of 143 failures grouped into 19 categories.

The failures include things like images without alt attributes, or with the wrong alt attributes, and blank link text. We also put in a number of things that we thought testing tools probably wouldn’t be able to detect, but are also accessibility issues. Things like flashing content that didn’t carry a warning, or plain language not being used in content.

We knew there was no way we could put in every potential accessibility barrier, but we wanted to have enough on the page so that we could adequately test how useful the tools were.

We then ran the tools against the page, to find out how many of the failures they would pick up and how many they would miss.

You can see our findings in detail here. Here are the main things we discovered:

Lots of the barriers weren’t found by any of the tools

We found that a large proportion of the barriers we created weren’t picked up by any of the 10 tools we tested – 29% in fact.

Of the 143 barriers we created, a total of 42 were missed by all of the tools we tested. The ones that were missed included barriers such as italics used on long sections of text, tables with empty cells and links identified by colour alone.

Even when barriers were found, the error reporting process wasn’t always clear-cut. Sometimes the tools would show a warning or call for manual inspection, without explicitly saying there was an error.

There is a huge range in the effectiveness of the tools

We also found that some of the tools picked up more errors than others.

If we only count error messages and warnings, then Tenon picked up the most barriers – it found 37% of them. If we also count manual inspection prompts, then Asqatasun was the most effective – it found 41% of the barriers.

At the other end of the range, Google Developer Tools, which is quite a popular tool, only picked up 17% of the barriers.

We found that using tools in combination could help you pick up more barriers, but doing this can be harder and less cost-effective for teams.

The effectiveness of the tools is just one of the things teams need to consider

We found a big range in terms of the effectiveness of the tools. But, as well as effectiveness, we also know that there are other considerations teams will take into account when deciding whether or not to use a tool, and which tool to use.

We know that the tools have to be easy to set up and run. And the results they give have to be clear and easy to act on. As well as being used by developers they may be used by non-technical people in teams.

There are other technical considerations to take into account too. For example, some tools might not work on password-protected pages. And some might not test on mobile pages.

As part of our work, we gathered contextual information about the tools to help teams make a decision on which ones suited them best.

How best to use automated tools

Our opinion of automated testing tools is the same after the audit as it was before. We think they are very useful and should definitely be used by teams to pick up issues. But also that they cannot be relied on, by themselves, to check the accessibility of a website. They are most effective when combined with manual testing.

Our research backs this up. While the tools picked up the majority of the accessibility barriers we created – 71% – there was a large minority that would only have been picked up by manual checking.

For the most effective accessibility testing, we advise teams to combine automated tool testing with manual checking, an accessibility audit and user testing.

We hope that our result pages will help teams pick a tool that best meets their needs. And will also encourage tool creators to better document what the tools can and can't do.

Follow Mehmet on Twitter and don't forget to sign up for email alerts.