@testcafe-community/axe does not report violations that other tools do

Question

I am exploring tools I can use for automated Accessibility Testing and wanted to try axe-core with TestCafe. I am an advocate of TestCafe, I love that is a lightweight tool and doesn't have dependencies on WebDriver. The docs are great and the scripting is easy.

I have however found that @testcafe-community/axe and its predecessor axe-testcafe do not report all violations while axe-core with selenium and axe-webdriverjs do. For example, running with axe-webdriverjs, I have the following code and resulting output showing the violations of a localhost page I am checking -

Code:

var AxeBuilder = require('axe-webdriverjs');
var WebDriver = require('selenium-webdriver');

const path = require('path');
process.env.PATH = path.resolve(`__dirname/../WebDriver/bin/Firefox/0.29.1`) + ':' + process.env.PATH;

var driver = new WebDriver.Builder()
  .forBrowser('firefox')
  .build();

driver
  //.get('https://dequeuniversity.com/demo/mars/')
  .get('http://localhost:3000')
  .then(function() {
    AxeBuilder(driver).analyze(function(err, results) {
      if (err) {
        // Handle error somehow
      }
      console.log(results.violations);
    });
  });

Output:

> [email protected] test1 /Users/nabeen.jamal/gitlab.com/notifications-service/text-messaging-application/tma-test/app/axe-webdriverjs-tests
> node test/axe.test1.js

[
  {
    description: 'Ensures all page content is contained by landmarks',
    help: 'All page content must be contained by landmarks',
    helpUrl: 'https://dequeuniversity.com/rules/axe/3.5/region?application=webdriverjs',
    id: 'region',
    impact: 'moderate',
    nodes: [ [Object], [Object] ],
    tags: [ 'cat.keyboard', 'best-practice' ]
  }
]

Using @testcafe-community/axe and following the 'How to use' on the project github page (https://github.com/testcafe-community/axe), I have the following code and resulting output which shows no violations of the same localhost page.

Code:

import { checkForViolations } from '@testcafe-community/axe';

fixture `TestCafe tests with Axe`
    //.page `http://example.com`;
    .page `http://localhost:3000`;

test('Automated accessibility testing', async t => {
    // do stuff on your page
    await checkForViolations(t);
});

Output:

nabeen.jamal@DEM-C02DFG1UMD6M axe-testcafe-tests % npx testcafe --config-file cfg/testcaferc.json chrome src/test1.js 
(node:88006) ExperimentalWarning: Conditional exports is an experimental feature. This feature could change at any time
(node:88006) ExperimentalWarning: Package name self resolution is an experimental feature. This feature could change at any time
 Running tests in:
 - Chrome 90.0.4430.212 / macOS 10.15.7

 TestCafe tests with Axe
 ✓ Automated accessibility testing


 1 passed (0s)

As the output shows, the @testcafe-community/axe test passes and shows no violations while the axe-webdriverjs (and axe-core with selenium) shows the violation about "all page content contained by landmarks".

Is this a limitation in @testcafe-community/axe, or do you have to specify the rules in the options parameter of axe.run for it to carry out the checks on the rendered content of the loaded page?

Answer 1

The documentation for axe-core states that you need to specify which rules you intend to test against using axe.run options.

Landmarks are discussed in WCAG 1.3.6., which is a "Level AAA" item. It appears that axe-core is only capable of testing against "Level A" and "Level AA." In your example, the item is not listed by the tool as a WCAG failure, but rather a best-practices recommendation. This may be why it isn't showing up in your other tools.

If you can easily implement this recommendation, then I'd say go ahead and do it. If not, I wouldn't let something like this stop my code from going into production. Landmarks are nice-to-have, but it's far more important that you meet all "Level A" requirements and as many "Level AA" requirements as you reasonably can.

It's worth noting that any automated accessibility testing tool is nothing more than a starting point for manual evaluation. These tools often generate tons of false positives (and sometimes miss important things!) because it's often not possible to algorithmically determine whether something is genuinely useful to human visitors.

I've also seen pages/apps that pass automated tools with no errors (Wave, Axe, etc.), but they are completely impossible to use with assistive technology.

Answer 2

I'm not 100% certain how, but my tests using axe-testcafe and @testcafe-community/axe are now showing the violation -

 Running tests in:
 - Chrome 90.0.4430.212 / macOS 10.15.7

 TestCafe tests with Axe
 ✓ Verify Welcome Page loads properly
 ✖ Automated accessibility testing

   1) AssertionError: 1 violations found:
      1) All page content must be contained by landmarks
      * https://dequeuniversity.com/rules/axe/3.5/region?application=axeAPI
      * cat.keyboard, best-practice
      * moderate
      * region
          "#global-cookie-message"
        ".app-phase-banner"
      : expected false to be truthy

      Browser: Chrome 90.0.4430.212 / macOS 10.15.7

         67 |const checkForViolations = async (t, context, options) => {
         68 |    const { error, results } = await axeCheck(t, context, options);
         69 |
         70 |    await t.expect(error).notOk();
         71 |
       > 72 |    await t.expect(results.violations.length === 0).ok(createReport(results.violations));
         73 |}
         74 |
         75 |
         76 |module.exports = {
         77 |    runAxe,

         at checkForViolations
      (/Users/nabeen.jamal/gitlab.com/notifications-service/text-messaging-application/tma-test/app/axe-testcafe-tests/node_modules/@testcafe-community/axe/index.js:72:53)



 1/2 failed (1s)

I didn't have to specify the rules in the options parameter to axe.run - I did do so, but with or without, the violation does now get reported.

I did however uninstall and reinstall the node packages and I think the version of axe-core is different to what I had previously. Here are my dependencies and the versions in my package.json that work for me -

{
  "name": "axe-testcafe-tests",
  "version": "1.0.0",
  "description": "axe-core and TestCafe testware to cover Accesibility Testing of the TMA App",
  "main": "index.js",
  "dependencies": {
    "testcafe": "^1.14.2"
  },
  "devDependencies": {
    "@testcafe-community/axe": "^3.5.0",
    "axe-core": "^3.5.5",
    "axe-testcafe": "^3.0.0"
  },
<-- snip -->

Thank you again @Josh for your help. Perhaps this might help others.