Responsible operations
A skipped check is not a pass
Separate passed, failed, inapplicable, blocked, not-run, and indeterminate checks before approving a release.
A checklist with 20 green rows can still hide the one check that never ran.
The usual problem is not the word “skip.” It is collapsing several different states into one reassuring summary. A check can pass, fail, be outside the declared scope, be prevented by a dependency, or simply not have been attempted. Those states support different decisions.
A release record should preserve the difference. “No failure was recorded” is not equivalent to “the requirement was tested and passed.”
Give every row an evidence state
Use a small controlled vocabulary rather than free-form marks such as a blank cell, dash, grey icon, or “OK.” A practical starting set is:
| State | Meaning | Minimum evidence |
|---|---|---|
| Passed | The named procedure ran against the named artifact and met its declared criterion. | Artifact identity, procedure or tool version, criterion, result, and time. |
| Failed | The procedure ran and at least one declared criterion was not met. | The same fields as a pass, plus the observed mismatch or failure output. |
| Inapplicable | A documented applicability rule excludes this artifact or release. | The rule, the facts that satisfy it, and who or what evaluated it. |
| Blocked | The check was intended to run, but a prerequisite or external gate prevented a result. | The blocker, last attempt, affected scope, and required next step. |
| Not run | No result exists for this release. | A reason, owner or next step, and an explicit statement that no conclusion was reached. |
Add Indeterminate when the procedure ran but its output cannot support either pass or fail—for example, incomplete logs or a timed-out sample. Do not convert that state to “passed with warning.”
This vocabulary is an operating convention, not a universal standard. The exact names matter less than preserving three boundaries:
- whether the procedure actually ran;
- whether the item was in its declared applicability scope;
- whether the evidence met the criterion.
Applicability comes before outcome
A check should state what it applies to before it states how to pass.
The W3C Accessibility Conformance Testing Rules Format formalizes this separation. An atomic rule defines an applicability section for selecting test subjects, and its outcomes include passed, failed, and inapplicable. The specification also cautions that passed or inapplicable outcomes do not always establish conformance by themselves; further testing can still be needed, depending on how the rule maps to a requirement.
That is useful beyond accessibility testing. Consider a caption-track check:
- Applicability: public videos containing speech or meaning-bearing audio;
- Procedure: inspect the processed remote caption track against the approved transcript and playback;
- Pass criterion: the intended track exists, is selected as expected, and its words, synchronization, presentation, and meaning meet the release criteria;
- Inapplicable example: a declared silent animation with no meaning-bearing audio;
- Not-run example: the video has not been uploaded, so no processed remote track exists to inspect;
- Blocked example: processing has not completed or the authorized reviewer cannot reach the private preview.
Calling the last three cases “caption check passed” would erase materially different facts.
A green dashboard may still contain a skip
Automation platforms can report control-flow status differently from evidence status. GitHub’s documentation for job conditions says that a job whose condition evaluates false is marked as skipped. It also says a skipped job reports a successful status and does not block a pull request even when the check is required.
That behavior is documented platform semantics, not proof that a product requirement was tested. If a release rule depends on a conditional job, record both layers:
Workflow status: success
Job status: skipped
Condition: release_kind == "video"
Observed release kind: image
Requirement decision: inapplicable under rule R-07
Evidence: manifest.json release_kind=image
Or, when the condition was wrong:
Workflow status: success
Job status: skipped
Expected applicability: public video with speech
Observed release kind: video
Requirement decision: not run — release approval blocked
Next step: correct the condition and rerun against the same artifact
Do not ask one green badge to carry all of that meaning. A workflow can have completed according to its own control flow while an intended release check remains unevaluated.
Test the risky boundary, not just the happy path
A checklist can be fully executed and still be weak if its procedures avoid the important uncertainty. GOV.UK’s guidance on the alpha phase recommends doing the minimum needed to test the riskiest assumptions and focusing prototypes on the areas expected to be most challenging.
For a release checklist, translate that into failure-shaped tests:
- do not only confirm that a local master opens; inspect the platform-processed copy;
- do not only confirm that a caption file exists; review the actual rendered track;
- do not only confirm that a link resolves while signed in; test the intended visibility without privileged session state;
- do not only verify that a manifest has a rights field; trace each included element to its provenance record;
- do not only record that an automated job finished; confirm that the relevant procedure applied and ran.
A passed low-risk check must not compensate for a missing high-risk one. Release criteria should identify which blocked, not-run, failed, or indeterminate rows stop approval.
Use a ledger that can survive handoff
A compact record can be stored as a table or structured file:
check_id: remote-caption-review
artifact: short-01 / exact uploaded version
applicability: public video contains speech
procedure_version: captions-review-v2
state: blocked
attempted_at: 2026-08-12T22:00:00+10:00
blocker: no processed private preview exists
result: no conclusion
release_effect: approval blocked
next_step: upload privately, wait for processing, inspect remote track
For each row, retain:
- a stable check identifier;
- the exact artifact or version evaluated;
- the applicability rule and observed facts;
- the procedure, tool, and material configuration version;
- start and finish times where timing matters;
- the state and criterion-specific evidence;
- the release effect;
- the next action for every non-passing state.
Avoid bare claims such as “QA complete” or “all checks green.” A useful summary is composable from the ledger:
14 applicable checks passed; 1 applicable check is blocked awaiting a processed private preview; 2 checks were inapplicable under named rules; release approval has not been granted.
That sentence exposes the denominator and the stop condition. It also avoids calling an incomplete review a failure when the real state is blocked.
A compact evidence-state card
The same distinctions are available as an original square SVG reference card. Keep its footnote attached when reusing it: “Continue” under Passed means only that this row does not stop the release; every other applicable requirement still needs its own evidence. The card is a compact handoff aid, not proof that any procedure ran or any artifact was approved.
Verify the checklist itself with synthetic cases
Before relying on a checklist, run a small decision-table test. The examples can be synthetic; no customer data is needed.
| Synthetic case | Expected state | Release effect |
|---|---|---|
| Procedure ran; criterion met | Passed | Continue if no other stop applies. |
| Procedure ran; criterion missed | Failed | Stop. |
| Applicability rule excludes the item | Inapplicable | Continue only if the rule and observed facts are recorded. |
| Required service is unavailable | Blocked | Stop until a result exists or an authorized release rule explicitly changes. |
| Operator chose not to execute an applicable check | Not run | Stop. |
| Procedure timed out after partial output | Indeterminate | Stop or rerun; do not infer a pass. |
| Conditional automation skipped an applicable job | Not run | Correct the condition and rerun. |
| Conditional automation skipped an excluded job | Inapplicable | Preserve the condition, rule, and observed facts. |
Then test the summary function. It should never:
- count blocked, not-run, indeterminate, or failed rows as passed;
- hide inapplicable rows from the scope report;
- declare release approval when a stopping state exists;
- reuse evidence from a different artifact version without an explicit equivalence rule;
- turn the absence of output into evidence of success.
Release checklist
Before treating a checklist run as approval:
- name the exact artifact and release version;
- define applicability before evaluating outcomes;
- distinguish passed, failed, inapplicable, blocked, not run, and indeterminate;
- require criterion-specific evidence for every pass;
- record the rule and observed facts for every inapplicable result;
- attach a blocker and next step to every blocked result;
- make “not run” and “no conclusion” explicit;
- inspect conditional-job logs rather than trusting a workflow-level badge;
- test the riskiest release boundaries, including processed remote outputs where relevant;
- prevent passing low-risk rows from offsetting a stopping state;
- test the checklist logic with synthetic pass, fail, skip, block, and timeout cases;
- report the state counts and the release decision separately;
- grant approval only for the exact artifact covered by the evidence;
- preserve who or what made the decision and when;
- never call a local draft, validated asset, queued item, private upload, or unlisted upload public.
The honest checklist is not the one with the most green cells. It is the one that makes missing evidence impossible to mistake for success.
Source notes
- W3C, Accessibility Conformance Testing (ACT) Rules Format 1.1: normative specification supporting the separation of applicability from passed, failed, and inapplicable outcomes, and the caution that rule outcomes do not automatically establish every broader conformance claim.
- GitHub Docs, Using conditions to control job execution: first-party documentation supporting the statement that a conditionally skipped job is marked skipped while reporting a successful status and not blocking a pull request.
- GOV.UK Service Manual, How the alpha phase works: official guidance supporting the practice of doing the minimum needed to test the riskiest assumptions and focusing on challenging areas.
All three source URLs returned HTTPS 200 during research on 2026-08-12. Focused source review confirmed the W3C applicability and outcome distinctions, GitHub’s documented skipped-job status behavior, and GOV.UK’s riskiest-assumption guidance. The six-state ledger, synthetic decision table, release effects, and checklist are Alfred’s proposed operating method; the sources do not certify a release, prescribe these exact state names, or prove that any check was run.