Benchmark · Accessibility
Accessibility engine Tested against the W3C ACT Rules test cases
787 test cases from the W3C’s ACT Rules, each with a known answer, run through the engine behind the site’s accessibility audit, and through axe-core for reference. The engine got 741 (94.2%) of them right. Every miss and every false alarm is listed below.
Auric Artisan engine 1.0.0
axe-core 4.12.1
- Correct
- Missed a failure
- False alarm
- Can’t tell
- Test cases scored
- 787
- ACT rules covered
- 50 of 87
- Run date
Results
How often each engine was right
A miss is a test case that fails the rule but was reported as passing. A false alarm is a test case that passes, or to which the rule does not apply, reported as failing. Can’t tell means the engine flagged it for a person to check.
| Engine | Correct | Missed a failure | False alarm | Can’t tell | Share of test cases |
|---|---|---|---|---|---|
| Auric Artisan engine 1.0.0 | 741 94.2% | 0 | 3 | 43 | |
| axe-core 4.12.1 | 679 86.3% | 14 | 7 | 87 |
- Correct
- Missed a failure
- False alarm
- Can’t tell
Rule by rule
Where each engine stands on each rule
Each row is an ACT rule with its test cases. The engine columns show how many it got right, then its misses and false alarms.
Failures
Every test case the engine got wrong
3 test cases, each linked to its page on w3.org, where the expected answer and the reason are given.
| Test case | ACT rule | Expected | Engine said | axe-core said |
|---|---|---|---|---|
Passed Example 8 0b43ded6 | Iframe elements with identical accessible names have equivalent purpose | passed | failed | can’t tell |
Passed Example 4 d343bc6a | Element with aria-hidden has no content in sequential focus navigation | passed | failed | can’t tell |
Inapplicable Example 2 7d3d7214 | Audio or video element avoids automatically playing audio | inapplicable | failed | can’t tell |
Coverage
ACT rules this run does not score
37 of the 87 ACT rules have no engine rule mapped to them, so their test cases were not run. Several need a person to judge, such as whether a caption is accurate.
- ARIA global properties not used where prohibited
- Audio and visuals of video element have transcript
- Audio element content has text alternative
- Audio or video element that plays automatically has a control mechanism
- Audio or video element that plays automatically has no audio that lasts more than 3 seconds
- Content has alternative for visual reference
- Device motion based changes to the content can also be created from the user interface
- Device motion based changes to the content can be disabled
- Element in sequential focus order has visible focus
- Error message describes invalid form field value
- Focusable element has no keyboard trap
- Focusable element has no keyboard trap via non-standard navigation
- Focusable element has no keyboard trap via standard navigation
- Form field label is descriptive
- HTML element language subtag matches language
- HTML images contain no text
- HTML page language subtag matches default language
- HTML page title is descriptive
- Heading is descriptive
- Image accessible name is descriptive
- Image not in the accessibility tree is decorative
- Link in context is descriptive
- Link is descriptive
- Links with identical accessible names and same context serve equivalent purpose
- No keyboard shortcut uses only printable characters
- Summary element has non-empty accessible name
- Text content that changes automatically can be paused, stopped or hidden
- Video element auditory content has captions
- Video element content is media alternative for text
- Video element visual content has accessible alternative
- Video element visual content has audio description
- Video element visual content has strict accessible alternative
- Video element visual-only content has accessible alternative
- Video element visual-only content has audio track alternative
- Video element visual-only content has transcript
- Video element visual-only content is media alternative for text
- Zoomed text node is not clipped with CSS overflow
Method
How it was run
- The fixtures. The ACT Rules Community Group’s published test cases (1,211 cases for 87 rules), each marked passed, failed or inapplicable. The file’s SHA-256 is recorded.
- The mapping. An engine rule counts for an ACT rule when the axe-core rule of the same name lists it, axe-core’s own mapping, or through 6 aliases where the site’s rule does the same job under another name. It is an assumption, and the results show where it does not hold.
- The run. Each HTML test case of a covered rule was loaded in Chromium 147.0.7727.15 at 1280 by 800, and the engine ran only its mapped rules. The page’s outcome is failed if any of them reports a violation, can’t tell if one is incomplete, otherwise passed. axe-core 4.12.1 ran the same way with its mapped rules.
- The engine. Bundled fresh from the site’s source on the day of the run, 127 rules; the bundle published below is that exact file.
Limits
What this does not show
- The engine was corrected against these test cases after the run of , which got 568 of 789 right, with 44 misses and 132 false alarms (that run’s results stay published). This run shows the corrections hold on the cases they were made for. On pages the engine has never seen, it can do worse.
- Only the 50 covered ACT rules are scored. A good score here says nothing about criteria the engine does not test.
- ACT test cases are small, isolated examples. Real pages combine many patterns, and an engine can behave differently there.
- One browser, one window size, no interaction: a rule that needs a hover, a focus or a scroll is tested as the page first loads.
- The axe-core figures use the same method and axe-core’s own mapping. They are a reference point, not axe-core’s official ACT implementation report, which is measured differently.
- w3.org answers headless browsers with a bot check, so each page was fetched by Node and handed to the browser unchanged; once loaded, a test case’s own redirects were stopped.
Rerun it
Files and commands
| File | Contents | Size | SHA-256 |
|---|---|---|---|
| results.json | Every test case with its expected outcome and both engines’ answers, the mapping and the versions. | 429.1 kB | 0437f92faebc… |
| engine.js | The exact engine bundle that was tested. | 286.0 kB | 6969eaebb66d… |
| run.mjs | Reruns every test case and compares with the published outcomes. | 4.6 kB | 4c103ad41653… |
| SHA256SUMS.txt | The checksums of the files above. | 0.2 kB | 2b2f4a2af94e… |
The results are CC BY 4.0: share and adapt them, with credit to Auric Artisan. run.mjs is CC0 1.0. engine.js is this site’s engine at the tested version, shared so the run can be repeated; it is not licensed for other use.
In a folder with these files, with Node.js 18 or later:
npm install playwright [email protected]
npx playwright install chromium
node run.mjs
References
Sources
- W3C: ACT Ruleswww.w3.org
- ACT Rules test cases (testcases.json)www.w3.org
- W3C: Accessibility Conformance Testing (ACT) Rules Format 1.1www.w3.org
- axe-core 4.12.1github.com
Found a mistake? Write to us and we will correct it.