feat(runner)!: declare scenarios a CSMS is expected to fail

.github/workflows/ci.yml

@@ -65,6 +65,13 @@ jobs:
       - name: Driver scope follows the environment
         run: bun tests/driver-env-scope.ts
 
+      # In-process for the same reason: the rule that decides whether a red
+      # sweep ends the build is not reachable from a shell without running a
+      # scenario, and the rows that matter would each need a CSMS engineered to
+      # fail a chosen way.
+      - name: Exit-code rule holds
+        run: bun tests/expected-failure-standing.ts
+
       - name: Core names no CSMS
         run: bash tests/generic-core.sh
 
@@ -79,42 +86,78 @@ jobs:
       - name: Shell guards lint clean
         run: shellcheck tests/*.sh tools/*.sh
 
-  # Everything above is offline and proves the drivers type-check. The two e2e
-  # jobs prove they work: real containers, real OCPP-J frames. They are peers,
-  # and each answers a question the other cannot -- see e2e-citrineos below.
+  # Everything above is offline and proves the drivers type-check. This job
+  # proves they WORK: real containers, real OCPP-J frames. It runs once per
+  # bundled driver, and the two lanes are peers -- each answers a question the
+  # other cannot.
+  #
+  # SteVe is the CSMS the 47 scenarios were originally written against, so it
+  # is where a capability the harness LOST shows up first. Its scope table
+  # claims every scenario; a row that had to be demoted would mean the core
+  # dropped something. Without this lane that can rot green unnoticed.
   #
-  # This one: SteVe is the CSMS the 47 scenarios were originally written
-  # against, so it is where a capability the harness LOST shows up first. Its
-  # scope table claims every scenario; a row that had to be demoted would mean
-  # the core dropped something. Without this job that can rot green unnoticed.
-  e2e-steve:
+  # CitrineOS had no part in writing the scenarios, so it is where the core
+  # would be caught quietly re-acquiring a dependency on how SteVe happens to
+  # behave. A CSMS-neutrality claim that only one CSMS ever exercises is a
+  # claim, not a check. It carries one known finding, and carries it in the
+  # open: cert16-tc023-3-authorize-blocked fails deterministically (3 runs of
+  # 3) because CitrineOS answers Invalid where OCPP 1.6 requires Blocked. The
+  # row is kept red rather than demoted in the scope table, which tck/scope.ts
+  # forbids, and it is DECLARED in drivers/citrineos/expected.ts -- so the
+  # sweep reports it as EXPECTED FAIL and exits 0, a NEW red still fails the
+  # build, and the day upstream fixes the mapping that row fails as UNEXPECTED
+  # PASS until its entry is deleted. drivers/citrineos/README.md carries the
+  # full gap table.
+  #
+  # ONE JOB RATHER THAN TWO NEAR-IDENTICAL COPIES, and it is only legal because
+  # both lanes are blocking: a matrix carries one continue-on-error for the
+  # whole strategy, so the arrangement this replaced -- one lane muted, one not
+  # -- could not have been written here at all.
+  e2e:
     runs-on: ubuntu-latest
     needs: check
-    # ~10 min of sweep plus image pull, a cold SteVe boot and the one-off
-    # provisioning restart. Generous, but far below the 6h default: a wedged
-    # simulator container should fail the build, not occupy a runner all day.
+    strategy:
+      # NOT the default, and not a preference. fail-fast cancels the sibling on
+      # the first failure, which would destroy the exact property the two
+      # paragraphs above describe: each lane answers a question the other
+      # cannot, and a cancelled lane is a missing answer rather than a saved
+      # minute.
+      fail-fast: false
+      matrix:
+        driver: [steve, citrineos]
+    # ~10 min of sweep plus image pull, a cold CSMS boot and -- for SteVe --
+    # the one-off provisioning restart. Generous, but far below the 6h default:
+    # a wedged simulator container should fail the build, not occupy a runner
+    # all day.
     timeout-minutes: 45
     env:
-      CSMS_DRIVER: ./drivers/steve/index.ts
-      # localhost, not the compose service name: the runner reaches SteVe
+      CSMS_DRIVER: ./drivers/${{ matrix.driver }}/index.ts
+      OCPP_CP_IDS: CERTCP1,CERTCP2,CERTCP3
+      # localhost, not the compose service name: the runner reaches the CSMS
       # through the published port, while the simulator containers reach it by
-      # service name from inside the network.
+      # service name from inside the network. That asymmetry is why each driver
+      # has a separate setting for the two.
+      #
+      # Both are set in both lanes. Each driver reads only its own, so the
+      # unused one is inert, and spelling them out here beats a
+      # matrix.include that hides which settings exist at all.
       STEVE_URL: http://localhost:8180/steve/manager
-      OCPP_CP_IDS: CERTCP1,CERTCP2,CERTCP3
+      CITRINE_API_URL: http://localhost:8080
     steps:
       - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
         with:
-          # Neither job pushes anything, so the token has no reason to stay
-          # in .git/config where every later step -- install scripts, the
-          # sweep, the containers it spawns -- could read it.
+          # This job pushes nothing, so the token has no reason to stay in
+          # .git/config where every later step -- install scripts, the sweep,
+          # the containers it spawns -- could read it.
           persist-credentials: false
 
       # Started before the toolchain, and deliberately without --wait: pulling
-      # two pinned images and cold-booting SteVe (its healthcheck allows 150s
-      # for Flyway) shares nothing with installing bun, so the two may as well
-      # overlap. The blocking wait below is then usually already satisfied.
-      - name: Start SteVe
-        run: docker compose -f drivers/steve/compose.yaml up -d
+      # the pinned images and cold-booting the CSMS (SteVe's healthcheck allows
+      # 150s for Flyway; CitrineOS runs its migrations on a cold database)
+      # shares nothing with installing bun, so the two may as well overlap. The
+      # blocking wait below is then usually already satisfied.
+      - name: Start the CSMS
+        run: docker compose -f drivers/${{ matrix.driver }}/compose.yaml up -d
 
       - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
         with:
@@ -123,7 +166,7 @@ jobs:
       - run: bun install --frozen-lockfile
 
       - name: Wait for it to be healthy
-        run: docker compose -f drivers/steve/compose.yaml up -d --wait
+        run: docker compose -f drivers/${{ matrix.driver }}/compose.yaml up -d --wait
 
       - name: Seed the fixtures the scenarios assume
         run: bun bin/ocpp-tck.ts driver provision
@@ -143,20 +186,22 @@ jobs:
       # barely moves with runner speed. Across 3 lanes, core is ~3 min and all
       # 44 are ~10 min. Roughly 7 minutes buys back the 29 scenarios that core
       # leaves out -- on precisely the branch where a regression is introduced,
-      # and for a job whose entire reason to exist is that this driver cannot
+      # and for a job whose entire reason to exist is that these drivers cannot
       # rot green. Cheap coverage is not worth a stale signal.
       #
       # --retry-failed-isolated is not optional here: the runner warns that
       # parallel lanes are not fully isolated, so a bare --parallel turns
       # cross-lane interference into a red build. The retry re-runs only the
       # failures, sequentially, which is the mode the runner calls reliable.
-      # `bun run e2e`, not the two commands spelled out here. A full sweep is
-      # two invocations -- `all` is 44 scenarios and the authorize group
-      # (TC_023) sits outside it, see TODO(upstream) at the GROUPS registry in
+      #
+      # `bun run e2e`, not the two commands it wraps. A full sweep is two
+      # invocations -- `all` is 44 scenarios and the authorize group (TC_023)
+      # sits outside it, see TODO(upstream) at the GROUPS registry in
       # tck/main.ts -- and that fact used to live only in this file. Anyone
       # running `run-all` by hand got 44/47 and read "no failures", missing
       # exactly the three scenarios that prove `driver provision` seeded
-      # anything. One script means CI and a human run the same thing.
+      # anything. One script means CI and a human run the same thing, and it
+      # means the two lanes cannot drift apart either.
       - name: Conformance sweep
         run: bun run e2e
 
@@ -165,93 +210,10 @@ jobs:
         if: always()
         uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
         with:
-          name: ocpp-tck-steve-results-${{ github.run_id }}
-          path: results/
-          if-no-files-found: warn
-
-      - name: Tear down
-        if: always()
-        run: docker compose -f drivers/steve/compose.yaml down -v
-
-  # The peer of the job above, answering the question it cannot: CitrineOS had
-  # no part in writing the scenarios, so this is where the core would be caught
-  # quietly re-acquiring a dependency on how SteVe happens to behave. A
-  # CSMS-neutrality claim that only one CSMS ever exercises is a claim, not a
-  # check.
-  #
-  # continue-on-error, and NOT because the job is unreliable: it is expected to
-  # be red. Two of the 47 scenarios fail against the pinned CitrineOS, and both
-  # are kept red deliberately rather than demoted in the scope table --
-  # tck/scope.ts forbids demoting a row to make a red scenario go away, and
-  # these are findings about CitrineOS:
-  #
-  #   cert16-tc023-3-authorize-blocked   deterministic (3 runs of 3): CitrineOS
-  #                                      answers Invalid where OCPP 1.6 requires
-  #                                      Blocked.
-  #   cert16-tc044-2-firmware-download-failed
-  #                                      flaky (1 pass in 3), thinnest timing
-  #                                      margin of any scenario.
-  #
-  # So this job's value is the OTHER 45, and the artifact it uploads. Turn it
-  # blocking only once both rows are green -- i.e. once upstream fixes the
-  # Blocked mapping -- and not merely because it has been quiet for a while.
-  # drivers/citrineos/README.md carries the full gap table.
-  e2e-citrineos:
-    runs-on: ubuntu-latest
-    needs: check
-    continue-on-error: true
-    timeout-minutes: 45
-    env:
-      CSMS_DRIVER: ./drivers/citrineos/index.ts
-      # localhost, not the compose service name -- same asymmetry as the SteVe
-      # job: the runner reaches the message API through the published port,
-      # while the simulator containers reach the OCPP endpoint by service name
-      # from inside the network.
-      CITRINE_API_URL: http://localhost:8080
-      OCPP_CP_IDS: CERTCP1,CERTCP2,CERTCP3
-    steps:
-      - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
-        with:
-          persist-credentials: false
-
-      # Before the toolchain and without --wait, for the same reason as the
-      # SteVe job: pulling the images and running the migrations on a cold
-      # database shares nothing with installing bun.
-      - name: Start CitrineOS
-        run: docker compose -f drivers/citrineos/compose.yaml up -d
-
-      - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
-        with:
-          bun-version: ${{ env.BUN_VERSION }}
-
-      - run: bun install --frozen-lockfile
-
-      - name: Wait for it to be healthy
-        run: docker compose -f drivers/citrineos/compose.yaml up -d --wait
-
-      - name: Seed the fixtures the scenarios assume
-        run: bun bin/ocpp-tck.ts driver provision
-
-      # Same reasoning as the SteVe job: a verb nothing runs is a verb that rots.
-      - name: Environment matches the scenarios
-        run: bun bin/ocpp-tck.ts driver verify
-
-      # `bun run e2e`, exactly as the SteVe job runs it -- both sweeps, one
-      # script, so the two jobs cannot drift apart and neither can drift from
-      # what a human runs by hand. Seven scenarios are NOT APPLICABLE here and
-      # never start a container, so this is cheaper than SteVe's rather than
-      # doubling the bill.
-      - name: Conformance sweep
-        run: bun run e2e
-
-      - name: Upload results
-        if: always()
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
-        with:
-          name: ocpp-tck-citrineos-results-${{ github.run_id }}
+          name: ocpp-tck-${{ matrix.driver }}-results-${{ github.run_id }}
           path: results/
           if-no-files-found: warn
 
       - name: Tear down
         if: always()
-        run: docker compose -f drivers/citrineos/compose.yaml down -v
+        run: docker compose -f drivers/${{ matrix.driver }}/compose.yaml down -v

AGENTS.md

@@ -34,6 +34,7 @@ bun run check:driver:steve
 bun run check:driver:citrineos
 bun run check:driver:citrineos-v1     # the same driver's other release line
 bun tests/driver-env-scope.ts
+bun tests/expected-failure-standing.ts
 ```
 
 then `bun run verify` once before committing.
@@ -79,9 +80,14 @@ There is no unit-test framework and no `*.test.ts`. `tests/` holds offline
 guards, each with a header stating the property it protects. `bun run test`
 chains them — note `bun test` is Bun's own runner and finds nothing here.
 
-Shell is the default; `tests/driver-env-scope.ts` is TypeScript because the
-property it asserts (a driver's declarations follow the env they are *resolved*
-with) is unreachable through the CLI, which can only ever pass `process.env`.
+Shell is the default, and the two TypeScript ones are TypeScript because what
+they assert is unreachable through the CLI. `driver-env-scope.ts`: a driver's
+declarations follow the env they are *resolved* with, where the CLI can only
+ever pass `process.env`. `expected-failure-standing.ts`: the rule that decides
+whether a red sweep ends the build, which from a shell would cost a container
+per row — and, for the rows that matter, a CSMS engineered to fail a chosen
+scenario a chosen way. `tck/standing.ts` is a module of its own so that guard
+can be a table.
 
 ## Two boundaries the guards enforce
 

CONTRIBUTING.md

@@ -173,6 +173,52 @@ answer. If you read those fields yourself rather than through the runner, use
 `driverScope(module, env)` / `driverCapabilities(module, env)` from
 `open-ocpp-tck/driver` instead of narrowing the union by hand.
 
+## Expected failures
+
+The scope table says what your CSMS **cannot drive**. This says what it drives
+and **gets wrong**. Keeping them apart is the point: a known-red scenario keeps
+its `DRIVABLE` row, still starts a container and still prints `FAIL` — what it
+stops doing is ending the build.
+
+```ts
+expectedFailures: {
+  "cert16-tc023-3-authorize-blocked": {
+    reason: "…the mechanism, cited, the way a scope row's is…",
+    finding: "…an upstream issue, or the row of your README's gap table…",
+  },
+},
+```
+
+Same placement and same resolution rules as `scope`: on the module, read with
+no credentials, and free to be a function of the environment
+(`driverExpectedFailures(module, env)` if you read it yourself).
+
+**A declared scenario that passes fails the sweep**, as `UNEXPECTED PASS`, and
+that half is not an inconvenience — it is what the mechanism is for. Without it
+the list only ever grows, and an entry outlives the defect it documents. When
+one fires, delete the entry or re-word it to say what is still true.
+
+**An entry excuses an answer, never a crash.** A declared scenario that `ERROR`s
+— container never started, bounded wait gave up, driver threw — still fails the
+sweep, reported as `DECLARED, BUT ERRORED`. It never got an answer out of the
+CSMS, so it cannot be the finding your `reason` describes, and a job that went
+green on it would be blind to exactly the kind of breakage it exists to catch.
+Your entry is probably still good; the crash is the new thing.
+
+Three rules, and they are the difference between a reviewed list and a mute:
+
+- `reason` names the mechanism. "Known red" is the observation being explained,
+  not the explanation.
+- `finding` says where to go and read about it. `check-driver` rejects an empty
+  one, because a known-red nobody can look up is a claim nobody can review.
+- **Never add an entry to quiet a flake.** There is no "expected flaky" status,
+  deliberately: a scenario that sometimes passes has a timing bug, and it will
+  be reported as an `UNEXPECTED PASS` the first time it does. Fix the scenario.
+
+`check-driver` also rejects an id that is stale, or that your scope table calls
+`NOT_APPLICABLE` — a scenario that never starts can neither fail as declared nor
+ever pass and delete the entry.
+
 ### Two worked examples
 
 Two drivers ship here, and they are worth reading together because they solve
@@ -188,8 +234,9 @@ For the mechanics of the contract — the operation switch, the record queries,
 the bootstrap verbs — either one works as a model. For **what to do when your
 CSMS cannot do something**, read `drivers/citrineos/`: it is the one with gaps,
 so it is the one that shows a scope table with `NOT_APPLICABLE` rows each citing
-the endpoint that does not exist, a `records` object that omits `reservations`
-outright rather than faking it, and a
+the endpoint that does not exist, an `expectedFailures` list with a row that
+stays `DRIVABLE` and red on purpose, a `records` object that omits
+`reservations` outright rather than faking it, and a
 [README](drivers/citrineos/README.md) whose gap table names a source fact per
 row.
 

README.md

@@ -174,7 +174,39 @@ the check could not be evaluated" is a real outcome:
   because a driver answered with `unverifiable("<why>")` instead of inventing a
   value. Exit code 0.
 
-Only `FAIL` and `ERROR` exit non-zero.
+`FAIL` and `ERROR` exit non-zero — unless your driver said so first.
+
+## Expected failures
+
+A CSMS you know gets one scenario wrong should not cost you the signal from the
+other 46. Declare it, and the sweep keeps running it, keeps printing `FAIL`,
+and stops failing the build for it:
+
+```ts
+export const csmsDriver: CsmsDriverModule = {
+  expectedFailures: {
+    "cert16-tc023-3-authorize-blocked": {
+      reason: "…the mechanism, cited…",
+      finding: "…where the finding is written down…",
+    },
+  },
+};
+```
+
+The scope row stays `DRIVABLE`. Demoting it to `NOT_APPLICABLE` instead would
+convert a finding about your CSMS into a silence about the harness, and
+`check-driver` rejects an id that is declared both ways.
+
+**A declared scenario that passes fails the sweep**, reported as
+`UNEXPECTED PASS`. That is the half that matters: it is how an entry gets
+deleted the day upstream fixes the defect, instead of outliving it. There is
+deliberately no "expected flaky" — a scenario that sometimes passes has a
+timing bug, and the bug is the thing to fix.
+
+**A declared scenario that `ERROR`s also fails the sweep**, as
+`DECLARED, BUT ERRORED`. An entry excuses what a CSMS *answers*; an `ERROR` is
+the scenario never getting an answer, and a green build there would be blind to
+the kind of breakage the mechanism exists to catch.
 
 ## Environment