JUnit report

@surea11y/core/junit renders a scan result as JUnit XML, the test report format CI dashboards read natively: GitLab's merge request test widget, Azure DevOps' Tests tab, Jenkins and CircleCI.

const { renderJunitReport } = require('@surea11y/core/junit');
const { runDomRulesInPage } = require('@surea11y/core');
 
const result = runDomRulesInPage(url, null, { profile: 'wcag22-aa' }, null);
require('fs').writeFileSync('surea11y.junit.xml', renderJunitReport(result));

renderJunitReport(result, options) is a pure function: it returns the XML as a string and never touches the filesystem. It works just as well on a result saved as JSON earlier, such as the output of surea11y scan <target> --json from @surea11y/cli. From the command line there is no need to call it: surea11y scan <target> --junit a11y.junit.xml (CLI 1.1.0 or later) writes the report itself, with the default options plus the baselineEntries of its --baseline file. See CI integration for GitLab and Azure DevOps recipes.

Shape

One <testsuite> per WCAG Success Criterion, one <testcase> per rule mapped to it. Scanning this page with the call above:

<!doctype html>
<html lang="en">
<head><title>Home</title></head>
<body><main><h1>Welcome</h1><img src="a.png"></main></body>
</html>

gives:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="surea11y" tests="14" failures="1" errors="0" skipped="4" time="0">
  <testsuite name="WCAG 1.1.1" tests="1" failures="1" errors="0" skipped="0" time="0">
    <properties>
      <property name="wcagCriterion" value="1.1.1"/>
      <property name="title" value="Non-text content: text alternatives"/>
      <property name="wcagLevel" value="A"/>
      <property name="criterionOutcome" value="fail"/>
      <property name="engine" value="a11ycore"/>
      <property name="schemaVersion" value="1.0.0"/>
      <property name="wcagVersion" value="2.2"/>
      <property name="profile" value="wcag22-aa"/>
      <property name="layout" value="false"/>
      <property name="locale" value="en"/>
      <property name="url" value="https://example.test/"/>
    </properties>
    <testcase classname="wcag-1.1.1" name="img-alt-present" time="0">
      <failure type="fail" message="1 failing occurrence: This &lt;img&gt; has no alt attribute.">- This &lt;img&gt; has no alt attribute. Add an alt attribute that says what the image shows, or alt=&quot;&quot; if the image is purely decorative.
  selector: html &gt; body &gt; main &gt; img
  html: &lt;img src=&quot;a.png&quot;&gt;
help: https://www.w3.org/WAI/WCAG22/Understanding/non-text-content.html</failure>
    </testcase>
  </testsuite>
  <!-- 10 more suites trimmed: 13 tests, 4 of them skipped (manual review) -->
</testsuites>

A green suite named for a criterion means the rules that ran for it reported no failure. It doesn't mean the criterion was fully checked, or met: most criteria have parts no rule covers (see WCAG mapping). Its criterionOutcome property says so: cantTell where the rules passed but check only part of the criterion, which today is every criterion.
Suites follow the criterion, because that is what people track, and testcases follow the rule rather than the occurrence, so a defect repeated forty times on a page is one failing test whose body lists all forty, and test counts stay stable between runs.
Names are the same in every locale. A suite is named WCAG <criterion> (WCAG 1.4.3), the suite of rules mapped to no criterion Other checks, a testcase by its rule id, and its classname is wcag-<criterion> or other, whatever engineOptions.locale the scan ran in. CI test history (GitLab, Jenkins, GitHub test reporters) is keyed on these names, so a scan read in French gives the same tests as one read in English. The criterion's title, which is translated, is the suite's title property (Contrast: minimum, Contraste (minimum)). The findings' own text (each occurrence's message and hint) is in the scan's locale; the words the report adds around it are English in every locale: the count that starts a message (1 failing occurrence: Cet élément <img> n’a pas d’attribut alt.), the selector:, html: and help: labels, and the <skipped> messages. Before core 1.12, a suite's name carried the criterion's title (WCAG 1.1.1 Non-text content: text alternatives), so the first report after upgrading shows every test once as new in a dashboard that tracked the old names, and their history starts again from there.
Suites come from each rule's own WCAG mappings, not from the composites, so every rule that ran is reported even when composites were excluded. The composite, when it ran, supplies the suite's title property and the criterionOutcome property, which counts the custom rules mapped to the criterion as it counts the built-in ones. A criterion checked by several composites (4.1.2: its accessible name and its ARIA validity) takes the worst of their outcomes, since one part failing fails the criterion, and the title their titles share (Name, role, value). A rule mapped to two criteria appears in both suites. A rule mapped to no criterion goes into a final Other checks suite with classname="other".
Suites follow the scan's WCAG level and version. A criterion the scan left out of rulesResults because of them gets no suite: an AAA criterion under engineOptions.wcagLevel: 'AA' (engine.wcagLevel), or 4.1.1 Parsing when includeRuleIds: ['duplicate-id'] ran under WCAG 2.2 (engine.wcagVersion), which removed it. A criterion the scan does report is kept, such as a rollup named by id above the level, a partly checked one, or 4.1.1 in a 2.2 scan that selected it. A rule mapped only to criteria left out goes into Other checks, and its failures are <skipped message="Not counted: WCAG 1.4.6 is outside the scan's target, WCAG 2.2 AA; 1 failing occurrence">, with the occurrences in <system-out>: the rule ran and failed, which the report shows, but an AA build does not fail on an AAA criterion. A rule that did not complete is still an <error>. A level the selection gives (runOnly.wcag.level, or level tags) is recorded as engine.wcagLevel too, and gates the suites the same way. A result that records neither engine.wcagLevel nor engine.wcagVersion, as an older engine's, keeps every criterion.
The run's engine, schemaVersion, wcagVersion, profile, optInRules, the rendering conditions from engine.environment (layout, and in a browser viewport as 1280x720, devicePixelRatio, colorScheme, forcedColors and fonts), locale and url are repeated as properties on every suite, each only when the result has it. A failure that comes and goes with the viewport width can be traced to the width the run used. The example above is a jsdom scan, which has no layout.
A page scanned at several viewport widths gives the same suite and test names at each width, and a CI test view may merge them. Write one file per width, and name or group the files by width in the CI's report settings; the viewport property records which width a file describes.
Suites are ordered by criterion, numerically (1.4.3 before 1.4.10), and testcases by rule id.
en301549 properties name the EN 301 549 clause that restates the criterion, where there is one and the scan asked for EN 301 549 clauses (see EN 301 549). Each other standard listed in the result's standards adds properties named by its key, one per requirement of it that the suite's rules map to: a pack's checklist item that groups the rule is listed by its id, <property name="acme" value="acme-images"/>. A standard that only a custom rule names adds no property.

A compact result (engineOptions.output.detail: 'findings') is read as the full result it stands for, so it is reported the same.

Cross-frame results

A cross-frame result (see Output schema) gives the page's suites first, then each frame's, named for the frame's path (Frame #checkout: WCAG 1.1.1, the same in every locale) and with a frame property holding it, and their testcases' classname starts with it (frame #checkout wcag-1.1.1), so a failure in a frame is not merged with the page's, by a CI that keys tests on suite names or on classname and name. A frame that did not answer is a suite Frame <path> with one skipped testcase, Not scanned: <error>: the frame was not checked, which is not a pass. The totals on <testsuites> count every frame.

Outcomes

Rule outcome	JUnit	Why
`fail`	`<failure type="fail">`, one line per failing occurrence (message, hint, selector, markup), then `help: <url>` when the rule has an `http(s)` help link (`meta.helpUrl`), or else its first WCAG criterion's Understanding document	The deterministic, gating case.
`cantTell`	`<skipped>` saying how many occurrences need manual review	JUnit has no "could not tell". Skipped surfaces it without turning a build red, the same line SARIF draws with `warning`.
`pass`	a bare `<testcase>`
`notApplicable`	left out	A page has hundreds; none says anything. `includeNotApplicable: true` adds them as `<skipped message="Not applicable">`.
a rule that did not complete (`cantTell` with no occurrences and an `error`: it threw)	`<error type="ruleError">` with the error, counted in `errors`	JUnit's element for a test that could not run. The page was not checked against the rule, which is neither a pass nor something to review.

A fail rule that also has cantTell occurrences reports the failures in <failure> and the undecided ones in <system-out>, where dashboards show test output. A skipped cantTell rule does the same.

The selector: line of an element inside a shadow tree names the shadow hosts first, then the element, joined with >>> : html > body > main > x-card >>> button. It is the same location SARIF and the HTML report give.

Options

Option	Default	Effect
`cantTellAs`	`'skipped'`	`'failure'` reports `cantTell` rules as `<failure type="cantTell">`, for a pipeline that must not pass while anything is undecided. Read in any case; another value throws a `TypeError`, as a misspelt one would quietly not gate.
`leanFailAs`	`'skipped'`	`'failure'` reports a rule whose undecided findings include one that leans to fail (its `lean`) as `<failure type="leanFail">`, listing those findings, each with why it likely fails (`likely fails:`), the lean's caution where it has one, and how a person confirms it (`how to check:`, the finding's `howToCheck` steps, then `fails when:`), which the report adds in Node to a result that comes without them; its other undecided findings stay as `cantTellAs` says. For a pipeline that gates on likely failures without gating on every question. A best-practice rule's too: such a rule, mapped to no WCAG criterion, never fails, and the facts it states lean fail without condition, so a skipped heading level (`heading-order`, an `<h1>` then an `<h3>`) is a `<failure type="leanFail">` in `Other checks`. A rule whose every criterion is outside the scan's target does not gate, as for a failure. Read in any case; another value throws a `TypeError`. `leaningFailures(result, { wcagOnly })`, from `@surea11y/core`, lists the findings it gates on, before the baseline.
`leanFailWcagOnly`	`false`	With `leanFailAs: 'failure'`, `true` gates only on the findings of rules mapped to a WCAG criterion in the scan's target, leaving out the best-practice rules' leans. A value other than `true` or `false` throws a `TypeError`.
`includeNotApplicable`	`false`	Include `notApplicable` rules as skipped tests.
`baselineEntries`	none	Entries from the baseline's `buildBaselineEntries()`. Fail occurrences recorded there are dropped, exactly as in SARIF. It takes the baseline file as saved (`JSON.parse` of it) or its `entries` list. A rule whose every failure is already known is `<skipped message="N known failures recorded in the baseline">`, not passing: it did not pass. An entry written by 1.13 under a reason code that changed in 1.14 still matches the finding under its new one. Under `leanFailAs: 'failure'`, a finding that leans fail and has an entry marked `lean: 'fail'` (`buildBaselineEntries(result, { leanFail: true })`) does not gate either, and the `<skipped>` message counts it among the `known likely failures recorded in the baseline`.
`packs`	none	The packs the scan ran (`engineOptions.packs`): with `leanFailAs: 'failure'`, a pack rule's likely failures give the steps of its own `meta.review`, as in the HTML report.
`name`	`'surea11y'`	The `name` of the root `<testsuites>`, for telling several pages' reports apart in one dashboard.

Determinism

The engine has no clock, and this report does not invent one: every time is "0", and a timestamp attribute appears on each suite only when the result carries one (engineOptions.timestamp). The same scan always renders byte-identical XML, so a report can be committed or diffed.

Text is escaped for XML, and characters XML 1.0 forbids even when escaped (most control characters, lone surrogates) are dropped, since markup captured from a page can contain them.