Test procedures

A scan decides what rules can decide. The rest of an assessment is a person's: the questions a scan leaves open (cantTell), and the parts of a criterion no rule tests at all. Since @surea11y/core 1.15.0, core says how to do that part too: a test procedure for every facet of each WCAG criterion its rules evaluate, and of the best practices, with the steps, the tools it needs, and what passes and what fails, in English, French, German, Spanish and Japanese. It is the same text a finding's “How to check” comes from, so a team's manual checks and its scans read the same.

Use them with no scan at all, for a manual audit; next to a scan, for what it leaves to a person; or to explain a criterion whose rules found nothing but don't cover all of it. Like the steps of a finding, procedures are guidance, not part of a finding's identity: their wording may improve in any release.

See them

These two pages are core's own output for WCAG 2.2 Level A and AA, regenerated with each release this site follows, and open from disk with nothing loaded from elsewhere:

• The reference: how to assess every criterion, facet by facet, under WCAG's principles and guidelines.
• The worksheet: each check once, with a result (Not checked, Passes, Fails, Does not apply) and notes to record, kept in your browser and downloadable as JSON or CSV.

Both pages are held to core's own rules, in jsdom and in Chromium, in light and dark mode, at the WCAG 2.2 AA profile: a skip link, one heading level at a time, labelled regions, real lists and fieldsets, visible focus, nothing in color alone, a forced-colors theme followed, a phone's width, and a printable layout.

What a procedure is

A criterion has facets, the test objectives a person checks to assess it. A facet a rule decides is full, one rules decide part of is partial, and one only a person can judge is manual. A facet has a general procedure: steps a person follows on a whole page, in a browser with its developer tools, a keyboard and, where it matters, a screen reader; then what they see if it passes and if it fails, naming the criterion it fails; and the tools it needs besides a browser:

• A technology facet is one test a rule makes on one kind of element (an <img> has an alt attribute). It has no procedure of its own: coveredBy names the broader facet whose procedure assesses it, so a checklist lists what a person does, once.
• A variant is the narrower question one rule asks about one element: the steps its undecided finding shows as howToCheck. A finding always shows its variant, never the general text, and the two never contradict each other. Every reason code a core rule can report as cantTell is under exactly one facet.
• What no criterion requires is kept apart, under best-practice, so a criterion's procedures never present advice as a failure.

Which criteria have procedures

Every criterion a core rule evaluates, in full, in part or as cantTell, has a procedure for every facet, the facets no rule checks included, since a person needs them to finish what the scan began: 36 of the 87 criteria of WCAG 2.2 at levels A, AA and AAA and 4.1.1 (removed in 2.2), with 170 facets, 56 procedures of their own and 114 technology facets covered by one of them. Such a criterion is complete. The other 51 criteria, which no rule evaluates (1.2.3 Audio Description or Media Alternative, say), are listed with their facets and no procedure: how to assess those by hand is outside core. The best practices have one for each of their 13 facets: landmarks, content in landmarks, headings, table markup, ARIA roles, focus order, access keys, skip links, names and labels, zoom, repeated names, new windows and downloads, and reduced motion.

Reading them

In Node they come from @surea11y/core, which reads the dictionaries from disk. Code that runs in a browser imports the same from @surea11y/core/procedures, with a From suffix (proceduresFrom, listProceduresFrom, procedureChecklistFrom, renderProceduresFrom, procedureLanguagesFrom), and passes the dictionaries it imports as @surea11y/core/procedures/<locale>: it needs no file system and reaches none of the rules.

const { proceduresFor } = require('@surea11y/core');
 
proceduresFor('1.4.12', { locale: 'fr' });
// { standard: 'wcag', id: '1.4.12', title: 'Espacement du texte', level: 'AA', complete: true,
//   rules: ['avoid-inline-spacing', 'text-spacing-content-loss'], facets: [ … ] }

import { proceduresFrom } from '@surea11y/core/procedures';
import en from '@surea11y/core/procedures/en';
import fr from '@surea11y/core/procedures/fr';
 
proceduresFrom({ en, fr }, '1.4.12', { locale: 'fr' });

Each facet carries its label, automation, the rules that claim it, its tools, its procedure ({ steps, passWhen, failWhen }, or null for a technology facet, with coveredBy) and its variants, each exactly the howToCheck of that rule's finding for that code. A text in another language than the one read (an earlier WCAG's own title, such as 2.1's “Parsing” for 4.1.1, or a text a locale lacks) says which, so a tool can mark it with lang, as WCAG 3.1.2 asks. Core's PROCEDURES.md on GitHub has the whole shape.

Listing them

const { listProcedures } = require('@surea11y/core');
 
listProcedures();                                         // WCAG 2.2, every level
listProcedures({ version: '2.1', levels: ['A', 'AA'] });  // WCAG 2.1 A and AA
listProcedures({ standard: 'best-practice' });
listProcedures({ levels: ['A', 'AA'], automation: ['manual', 'partial'] }); // what a scan leaves

WCAG's criteria are those the version has, at its levels: 2.0 and 2.1 list 4.1.1 as a Level A criterion titled “Parsing”, 2.2 doesn't. Given several filters, a facet must match each; a requirement's complete stays its own.

A checklist

procedureChecklist takes the same options and gives what a person does to assess the standard, once each: an item per general procedure, with the technology facets it also assesses in covers, and the rules behind it in rules.

const { procedureChecklist } = require('@surea11y/core');
 
procedureChecklist({ levels: ['A', 'AA'] }).items[0];
// {
//   key: '1.1.1/text-alternative-mechanism',
//   label: 'Non-text content that conveys something has a text alternative',
//   automation: 'partial',
//   wcag: '1.1.1',
//   requirements: ['1.1.1'],
//   covers: ['img-alt-attr-present', 'role-img-text-alternative-present', …],
//   rules: ['img-alt-present', …],
//   tools: [{ id: 'devtools', label: "The browser's developer tools" }],
//   procedure: { steps: ['…'], passWhen: '…', failWhen: '… which fails WCAG 1.1.1.' }
// }

Pages and worksheets

renderProcedures turns what listProcedures, procedureChecklist or proceduresFor gives into a document a person reads or works through: one self-contained HTML page, which opens from disk with no server, network, font or script from elsewhere, or Markdown. The same input and options always give the same page.

const { listProcedures, procedureChecklist, renderProcedures } = require('@surea11y/core');
const fs = require('node:fs');
 
// A reference: how to assess every WCAG 2.2 A and AA criterion.
fs.writeFileSync('wcag-22-aa.html', renderProcedures(listProcedures({ levels: ['A', 'AA'] })));
 
// A worksheet: each check once, with a result and notes to record.
fs.writeFileSync('wcag-22-aa-checklist.html', renderProcedures(procedureChecklist({ levels: ['A', 'AA'] })));
 
// The same in Markdown, for a repository or a wiki.
fs.writeFileSync('wcag-22-aa.md', renderProcedures(listProcedures({ levels: ['A', 'AA'] }), { format: 'markdown' }));

A list's page is a reference for the standard: its title and scope, what its results mean (the best practices count against no conformance, Level AAA is advisory), “Before you start”, a table of contents, and each requirement with its facets: what each checks, how much of it rules decide, the tools, the steps, what passes and what fails, and the rules that test it.

A checklist's page is a worksheet. Under each item a person records a result and notes; the page counts the results in a status message a screen reader announces, keeps them in the browser (localStorage, keyed by the standard, version and title), and downloads them as JSON, with each requirement's result rolled up from its items, or as CSV, which a spreadsheet opens with its accents. A requirement fails where one of its items fails, is open where one is not checked yet, is notApplicable where none applies, and passes otherwise.

{
  "title": "WCAG 2.2: manual checklist",
  "standard": "wcag",
  "version": "2.2",
  "items": [
    {
      "key": "1.1.1/text-alternative-mechanism",
      "requirements": ["1.1.1"],
      "outcome": "fail",
      "notes": "The logo has no alt."
    }
  ],
  "requirements": [{ "id": "1.1.1", "outcome": "fail" }]
}

A page's own words are in every language core ships, and so are the procedures, the facets' labels and WCAG's principles, guidelines and criteria, the names as each language's published translation of WCAG gives them. What is in another language than the page's is marked with it, so a screen reader reads each part in its language. fragment: true gives the content alone, in a <div class="surea11y-procedures"> with its headings from h1, for a documentation site that brings its own styles.

From the CLI

@surea11y/cli 1.5.0 writes the same pages with no code: surea11y procedures with no scan, and --manual-checks next to a scan, for the checks it leaves to a person.

# How to assess every WCAG 2.2 A and AA criterion, or a worksheet for a manual audit
$ surea11y procedures --wcag-level AA -o wcag-22-aa.html
$ surea11y procedures --wcag-level AA --checklist --title "Audit: example.com" -o audit.html
# The checks a scan leaves to a person, as a worksheet next to its report
$ surea11y scan https://example.com/ --wcag 2.2 --html report.html --manual-checks manual-checks.html

In the HTML report

The HTML report has a “What's left for a person” section: every requirement of the standard scanned with a check no rule decides in full, its outcome in the scan beside it, and, collapsed under it, those checks with their steps. A requirement whose rules all pass is met only once its checks pass too, and the report says so. renderHtmlReport(result, { manualChecks: false }) leaves the section out.

Uses

A manual audit, with no scan. An auditor works through the worksheet of the standard the audit is against, records each result and note, and downloads the JSON for the report:

const checklist = procedureChecklist({ version: '2.2', levels: ['A', 'AA'] });
fs.writeFileSync('audit-worksheet.html', renderProcedures(checklist, { title: 'Accessibility audit: example.com' }));

A test run's attachment. A Playwright test that scans a page can attach the checks its rules don't decide, so they sit next to the scan in the test report:

test('home page', async ({ page }, testInfo) => {
  // ... scan the page ...
  const left = procedureChecklist({ levels: ['A', 'AA'], automation: ['manual', 'partial'] });
  await testInfo.attach('manual-checks.html', {
    body: renderProcedures(left, { record: false }),
    contentType: 'text/html'
  });
});

A session with a screen reader user. Only what needs a screen reader, for a session with a person who uses one every day:

const sr = listProcedures({ levels: ['A', 'AA'], tools: 'screen-reader' });
fs.writeFileSync('screen-reader-session.html', renderProcedures(sr, { title: 'Screen reader session' }));

Documentation. One Markdown file per standard in a documentation site or a repository, regenerated when core is updated, so a team reads the same procedures its tools use. One criterion, where it is needed: a tool that shows a finding can show how to assess its whole criterion, with renderProcedures(proceduresFor('1.4.12', { locale: 'de' }), { fragment: true }).

A pack's standard

A pack's standard can say how a person assesses its own requirements, and every function above reads it given the packs: by its own numbers and groups, in its own language where it declares one. A requirement that restates a WCAG criterion takes that criterion's procedures, read as the requirement (“which fails EN 301 549 9.1.4.3”). Since core 1.16.0, a facet also lists the core rules the standard's mapping sends to what it assesses (mappedRules, and in a checklist item's mappedRules), so a requirement or test only core's rules check doesn't look unchecked.

const en301549 = require('@surea11y/pack-en301549');
 
proceduresFor('9.1.4.3', { packs: [en301549], standard: 'en301549', version: 'V4.1.1', locale: 'fr' });
relatedRequirements('1.4.3', { packs: [en301549] });
// [{ standard: 'EN 301 549', id: '9.1.4.3', version: 'V4.1.1', relation: 'restates', … }]
 
const options = { packs: [en301549], standard: 'en301549', version: 'V4.1.1', locale: 'fr' };
fs.writeFileSync('en-301-549.html', renderProcedures(procedureChecklist(options), { packs: [en301549] }));

The EN 301 549 pack 1.2.0 gives procedures for every chapter 9 clause whose criterion a core rule evaluates, in five languages, and the WAI-ARIA pack gives a checklist for each of its requirements, with the roles, states and values that are valid listed in full. A pack that gives no procedures changes nothing: WCAG's are the same with it as without it.