Benchmark · Accessibility

Accessibility engine Tested against the W3C ACT Rules test cases

787 test cases from the W3C’s ACT Rules, each with a known answer, run through the engine behind the site’s accessibility audit, and through axe-core for reference. The engine got 741 (94.2%) of them right. Every miss and every false alarm is listed below.

Auric Artisan engine 1.0.0

axe-core 4.12.1

  • Correct
  • Missed a failure
  • False alarm
  • Can’t tell
Test cases scored
787
ACT rules covered
50 of 87
Run date

Results

How often each engine was right

A miss is a test case that fails the rule but was reported as passing. A false alarm is a test case that passes, or to which the rule does not apply, reported as failing. Can’t tell means the engine flagged it for a person to check.

Outcomes by engine
EngineCorrectMissed a failureFalse alarmCan’t tellShare of test cases
Auric Artisan engine 1.0.0741 94.2%0343
axe-core 4.12.1679 86.3%14787
  • Correct
  • Missed a failure
  • False alarm
  • Can’t tell

Rule by rule

Where each engine stands on each rule

Each row is an ACT rule with its test cases. The engine columns show how many it got right, then its misses and false alarms.

Results for each ACT rule
ACT ruleCasesAuric Artisan engine 1.0.0axe-core 4.12.1
ARIA attribute is defined in WAI-ARIA 5f99a788/8all right8/8all right
ARIA required context role ff89c91515/15all right15/15all right
ARIA required owned elements bc4a752424/24all right23/241 missed
ARIA state or property has valid value 6a72812020/20all right19/201 can’t tell
ARIA state or property is permitted 5c01ea1717/17all right17/17all right
Audio element content has transcript 2eb176110/1111 can’t tell0/1111 can’t tell
Audio element content is media alternative for text afb42370/77 can’t tell0/77 can’t tell
Audio or video element avoids automatically playing audio 80f0bf86/81 false1 can’t tell4/84 can’t tell
Autocomplete attribute has valid value 73f2c23030/30all right30/30all right
Block of repeated content is collapsible 3e12e171/76 can’t tell0/77 can’t tell
Button has non-empty accessible name 97a4e11717/17all right17/17all right
Bypass Blocks of Repeated Content cf77f21311/132 can’t tell8/135 can’t tell
Document has a landmark with non-repeated content b40fd177/7all right4/73 can’t tell
Document has an instrument to move focus to non-repeated content ye5d6e1111/11all right7/112 missed2 can’t tell
Document has heading for non-repeated content 047fe01313/13all right9/132 missed2 can’t tell
Element marked as decorative is not exposed 46ca7f1010/10all right10/10all right
Element with aria-hidden has no content in sequential focus navigation 6cfa841514/151 false13/152 can’t tell
Element with lang attribute has valid language tag de46e41919/19all right19/19all right
Element with presentational children has no focusable content 307n5z1212/12all right12/12all right
Element with role attribute has required states and properties 4e8ab61616/16all right16/16all right
Form field has non-empty accessible name e086e52222/22all right22/22all right
HTML page `lang` attribute has valid language tag bf051a66/6all right6/6all right
HTML page has lang attribute b5c3f855/5all right5/5all right
HTML page has non-empty title 2779a51212/12all right12/12all right
Headers attribute specified on a cell refers to cells in the same table element a25f451919/19all right19/19all right
Heading has non-empty accessible name ffd0e91515/15all right15/15all right
Iframe element has non-empty accessible name cae7601111/11all right11/11all right
Iframe elements with identical accessible names have equivalent purpose 4b1c6c2322/231 false12/233 missed8 can’t tell
Iframe with interactive elements is not excluded from tab-order akn7bn1010/10all right9/101 missed
Image button has non-empty accessible name 59796f1212/12all right12/12all right
Image has non-empty accessible name 23a2a81818/18all right18/18all right
Important letter spacing in style attributes is wide enough 24afc21818/18all right16/182 false
Important line height in style attributes is wide enough 78fd322323/23all right23/23all right
Important word spacing in style attributes is wide enough 9e45ec1818/18all right15/183 false
Link has non-empty accessible name c487ae2828/28all right28/28all right
Links with identical accessible names have equivalent purpose b20e66219/2112 can’t tell10/2111 can’t tell
Menuitem has non-empty accessible name m6b1q388/8all right8/8all right
Meta element has no refresh delay bc659a1515/15all right15/15all right
Meta element has no refresh delay (no exception) bisz581313/13all right13/13all right
Meta viewport allows for zoom b4f0c31616/16all right14/162 missed
Object element rendering non-text content has non-empty accessible name 8fc3b61818/18all right18/18all right
Orientation of the page is not restricted using CSS transforms b33eff1313/13all right12/131 can’t tell
Role attribute has valid value 674b101111/11all right11/11all right
SVG element with explicit role has non-empty accessible name 7d67341010/10all right10/10all right
Scrollable content can be reached with sequential focus navigation 0ssw9k1515/15all right15/15all right
Table header cell has assigned cells d0f69e1616/16all right13/161 missed2 can’t tell
Text has enhanced contrast 09o5cg3534/351 can’t tell27/358 can’t tell
Text has minimum contrast afw4f73433/341 can’t tell26/348 can’t tell
Video element auditory content has accessible alternative eac66b64/62 can’t tell2/64 can’t tell
Visible label is part of accessible name 2ee8b83636/36all right31/362 missed2 false1 can’t tell

Failures

Every test case the engine got wrong

3 test cases, each linked to its page on w3.org, where the expected answer and the reason are given.

Test cases the engine got wrong
Test caseACT ruleExpectedEngine saidaxe-core said
Passed Example 8 0b43ded6Iframe elements with identical accessible names have equivalent purposepassedfailedcan’t tell
Passed Example 4 d343bc6aElement with aria-hidden has no content in sequential focus navigationpassedfailedcan’t tell
Inapplicable Example 2 7d3d7214Audio or video element avoids automatically playing audioinapplicablefailedcan’t tell

Coverage

ACT rules this run does not score

37 of the 87 ACT rules have no engine rule mapped to them, so their test cases were not run. Several need a person to judge, such as whether a caption is accurate.

Method

How it was run

  • The fixtures. The ACT Rules Community Group’s published test cases (1,211 cases for 87 rules), each marked passed, failed or inapplicable. The file’s SHA-256 is recorded.
  • The mapping. An engine rule counts for an ACT rule when the axe-core rule of the same name lists it, axe-core’s own mapping, or through 6 aliases where the site’s rule does the same job under another name. It is an assumption, and the results show where it does not hold.
  • The run. Each HTML test case of a covered rule was loaded in Chromium 147.0.7727.15 at 1280 by 800, and the engine ran only its mapped rules. The page’s outcome is failed if any of them reports a violation, can’t tell if one is incomplete, otherwise passed. axe-core 4.12.1 ran the same way with its mapped rules.
  • The engine. Bundled fresh from the site’s source on the day of the run, 127 rules; the bundle published below is that exact file.

Limits

What this does not show

  • The engine was corrected against these test cases after the run of , which got 568 of 789 right, with 44 misses and 132 false alarms (that run’s results stay published). This run shows the corrections hold on the cases they were made for. On pages the engine has never seen, it can do worse.
  • Only the 50 covered ACT rules are scored. A good score here says nothing about criteria the engine does not test.
  • ACT test cases are small, isolated examples. Real pages combine many patterns, and an engine can behave differently there.
  • One browser, one window size, no interaction: a rule that needs a hover, a focus or a scroll is tested as the page first loads.
  • The axe-core figures use the same method and axe-core’s own mapping. They are a reference point, not axe-core’s official ACT implementation report, which is measured differently.
  • w3.org answers headless browsers with a bot check, so each page was fetched by Node and handed to the browser unchanged; once loaded, a test case’s own redirects were stopped.

Rerun it

Files and commands

The benchmark’s files
FileContentsSizeSHA-256
results.jsonEvery test case with its expected outcome and both engines’ answers, the mapping and the versions.429.1 kB0437f92faebc…
engine.jsThe exact engine bundle that was tested.286.0 kB6969eaebb66d…
run.mjsReruns every test case and compares with the published outcomes.4.6 kB4c103ad41653…
SHA256SUMS.txtThe checksums of the files above.0.2 kB2b2f4a2af94e…

The results are CC BY 4.0: share and adapt them, with credit to Auric Artisan. run.mjs is CC0 1.0. engine.js is this site’s engine at the tested version, shared so the run can be repeated; it is not licensed for other use.

In a folder with these files, with Node.js 18 or later:

Terminal

npm install playwright [email protected]
npx playwright install chromium
node run.mjs

References

Sources

Found a mistake? Write to us and we will correct it.