EARL report

@surea11y/core/earl renders scan results as EARL 1.0 in JSON-LD — the vocabulary the W3C publishes for stating "this tool tested this thing and got this result," and the format the ACT Rules community group accepts as an implementation report.

const { renderEarlReport } = require('@surea11y/core/earl');
 
const report = renderEarlReport(result, {
  assertor: { name: 'surea11y', version: '1.16.1' },
  mode: 'earl:automatic'
});
 
fs.writeFileSync('earl.jsonld', JSON.stringify(report, null, 2));

What it is for

Two audiences, and they want the same document for different reasons.

An implementation report tells the ACT Rules community group how this engine behaves against their test cases, which is what gets an engine listed alongside the other implementations. Listing is not endorsement and the W3C does not verify the data — the accurate phrasing is "listed as an ACT implementation," never "W3C certified." This engine publishes two, both regenerated by scripts/act-report.js from the latest release: one from a scan in jsdom, as the CLI and test-matchers run the engine, with the assertor named “SureA11y”; and one from a scan in Chromium (--runner=chromium), as the browser bindings run it, with the assertor named “SureA11y in a browser” and the Chromium version in the report's description. A workflow in SureA11y/act-report republishes both every Monday, and is run by hand after each release.

A consumer gets an interchange format. EARL is what accessibility tooling reads when it has to combine results from more than one source — an automated scan and a manual audit, say, or several engines — because every assertion carries who asserted it and how. That is worth having whether or not anything is ever submitted anywhere.

How it differs from the other reporters

SARIF report and HTML report both carry violations only: they report fail and cantTell occurrences and drop everything else. EARL is the opposite. Every rule that ran becomes an assertion, pass and inapplicable included, because an implementation report is a claim about what the engine decided everywhere. A rule that stayed silent because it found nothing applicable is evidence, not noise — it is how a reader distinguishes "this engine checked and found nothing to check" from "this engine does not implement that rule at all." Each assertion is about one rule on one subject: a rule's passed is not a verdict on the WCAG criterion it maps to, most of which have parts no rule covers (see WCAG mapping).

Shape

The graph groups by subject rather than being a flat list of assertions:

{
  "@context": "https://www.w3.org/WAI/content-assets/wcag-act-rules/earl-context.json",
  "@graph": [
    {
      "@type": "TestSubject",
      "source": "https://example.test/",
      "assertions": [
        {
          "@type": "Assertion",
          "test": { "title": "img-alt-present", "isPartOf": ["WCAG2:non-text-content"] },
          "result": { "outcome": "earl:failed" },
          "assertedBy": {
            "@type": "Assertor",
            "name": "surea11y",
            "release": { "@type": "Version", "revision": "1.16.1" }
          },
          "mode": "earl:automatic"
        }
      ]
    }
  ]
}

• source is the scanned URL, or about:blank when a result carries none. When several results carry none, each is a subject of its own, about:blank#result-<n> by its position in the list (from 1), rather than all merging into one.
• test.title is the engine's own rule id. In ACT terms a rule is the procedure the implementation ran, which is exactly what a rule id names.
• test.isPartOf lists the Success Criteria that rule maps to, as WCAG2:<criterion-id>: each normativeMappings entry whose standard is WCAG or absent (as in SARIF) and that has no type. Omitted entirely for a rule claiming no criterion — aria-allowed-role, empty-heading and landmark-role-name-present are the engine's automatic rules in that position, and asserting an empty list would read as "maps to nothing we could find" rather than "deliberately maps to none." Such a rule can still fail (empty-heading fails a heading with no name): its assertion is then earl:failed with no isPartOf, a failure of the rule that claims no WCAG criterion failed.
• isPartOf follows the scan's WCAG level and version. A criterion the scan left out of rulesResults because of them is not listed, such as 1.4.6 (AAA) under engineOptions.wcagLevel: 'AA', or 4.1.1 when includeRuleIds: ['duplicate-id'] ran under WCAG 2.2, which removed it; a rule whose every criterion is left out has no isPartOf. The assertion itself stays, as it is about the rule. A criterion the scan reports is listed, such as a rollup named by id above the level, or 4.1.1 in a 2.2 scan that selected it, and so is every criterion of a result that records neither engine.wcagLevel nor engine.wcagVersion.
• assertedBy is always there unless you pass assertor: null: it defaults to surea11y and the engine version the results carry. mode appears only when you supply it.

Criterion ids are derived from the criterion's own title (Non-text Content → non-text-content). normativeMappings also carries Understanding-document references, which have a type, and other standards' entries (EN 301 549 clauses, a pack's requirements or checklist items), which name their own standard; both can share a requirement number with a criterion. Only entries whose standard is WCAG or absent and that have no type are read as Success Criteria. No conformance level is needed. isPartOf lists WCAG criteria only; other standards' entries are left out.

A compact result (engineOptions.output.detail: 'findings', see Output schema) is read as the full result it stands for, so it is reported the same.

Outcomes

earl:untested has no counterpart: a rule that did not run produces no result to assert on, so it contributes no assertion rather than an untested one.

cantTell does not cost conformance credit. ACT's own consistency rules allow an automated implementation to report "cannot tell" on some — though not all — examples and still count as consistent. What a partially consistent implementation may not do is produce a false positive: failing an example the rule says should pass, or that is inapplicable. That is the gate worth watching, and it is a property of the rules rather than of this reporter. As of the last full run the engine produces zero false positives across the 957 examples W3C publishes for its 60 matched rules — see ACT_RULE_MAPPING.md, and re-run scripts/act-testcase-check.js for the live figure rather than trusting this one indefinitely.

Several results, one report

renderEarlReport takes an array as readily as a single result, because a report covering many pages is the normal case:

renderEarlReport([homeResult, checkoutResult, searchResult], { assertor });

A cross-frame result from runa11yCoreAcrossFrames is taken as it is, alone or in the array: each frame that answered is one more result, and becomes a subject under its own URL. A frame below the top one whose URL does not tell it apart, because it is about:srcdoc or about:blank or another frame of the result has it too, is named by its URL and its path, the selectors SARIF and JUnit name it by, each encoded and joined with /: about:srcdoc#frame=%23pay, about:srcdoc#frame=%23widget/%23inner. Where even the path is shared, as for a binding's frames, which carry no selector, its place among the result's frames is added, from 1: about:srcdoc#frame=iframe&n=3. Anything else that is not a scan result, such as null or a string, throws a TypeError.

Results sharing a URL merge into one subject — a caller scanning the same page under different engineOptions is still describing one resource, and the context has no way to express two subjects with the same source. Where two results assert on the same rule for the same URL, the worse outcome is kept: earl:failed, then earl:cantTell, then earl:passed, then earl:inapplicable, so no order of the inputs can hide a failure. That includes the same page scanned at two viewport widths: EARL has no term for the conditions a page was rendered under, so a report built from several widths keeps, for each rule, a failure at any width. Render one report per width if the widths themselves matter.

Output is deterministic: subjects sort by source, assertions by rule id, and the same inputs produce byte-identical output in any order. The one exception is results without a URL, which are told apart by their place in the list (about:blank#result-2); give each result its URL when order should not matter. That is what makes a diff between two engine versions meaningful.

A scan of several profiles

In a result of several profiles (core 1.15.0 and later), a check's outcome is the worst its profiles give it, which can be a standard's failure of a finding WCAG only asks about. Each assertion takes the rule's own outcome, checkOutcome, instead: an attribute a role doesn't support, which WAI-ARIA fails and WCAG 4.1.2 asks about, is asserted earl:cantTell against WCAG2:name-role-value. So a standard's failure is never asserted as a WCAG one.

Options

See also

• Output schema — the result this reads
• ACT_RULE_MAPPING.md — which ACT rules the engine's rules correspond to
• API_STABILITY.md — what is covered by semver