From efad8857394fe11fe12899deae5584c6c849007e Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 14 Aug 2026 09:39:08 +0200 Subject: [PATCH 01/11] feat(runner): declare scenarios a CSMS is expected to fail A driver may now list scenarios it knows its CSMS fails, with the mechanism and where the finding is written down. The scope row stays DRIVABLE, the scenario still runs and still prints FAIL; what changes is the exit code. The vocabulary this replaces was `continue-on-error` on a whole CI job, which is job-granular where the intent is scenario-granular: a broken compose file, a failed provisioning step and a regression from one failure to twenty were all as green as the finding the job was muted for, so the job could not report the one thing it existed to report -- every other scenario. The list has to be able to shrink, or it is a quieter mute. So a declared scenario that PASSES fails the sweep, as UNEXPECTED PASS: that is how an entry gets deleted the day upstream fixes the defect, instead of outliving it. There is deliberately no "expected flaky" status -- a scenario that sometimes passes has a timing bug, and admitting the status would re-create the job-level mute one scenario at a time. Both directions read one predicate, effectivelyFailed(), extracted from the existing --retry-failed-isolated rule rather than restated: written twice they would drift, and the drift would be silent exactly where it hurts. check-driver rejects, offline, an entry that is stale, that contradicts a NOT_APPLICABLE scope row, or that carries an empty reason or finding. What it cannot check is whether the scenario still fails -- only a sweep answers that. `ocpp-tck run` honours the same list, so running one scenario by hand means what running it in the sweep means; without it `bun run e2e:smoke`, which names scenarios explicitly, would contradict `bun run e2e` on the same commit. The field is optional: a driver that declares nothing keeps the original rule, and driver-env-scope.ts asserts that resolving an absent list still yields undefined. --- CONTRIBUTING.md | 44 +++- README.md | 29 ++- VENDOR.md | 3 +- package.json | 4 + patches/tck/main.ts.patch | 427 ++++++++++++++++++++++++++++++++------ tck/driver.ts | 32 +++ tck/expected.ts | 107 ++++++++++ tck/index.ts | 1 + tck/main.ts | 331 +++++++++++++++++++++++++++-- tests/driver-env-scope.ts | 64 +++++- types/tck/driver.d.ts | 24 +++ types/tck/expected.d.ts | 79 +++++++ types/tck/index.d.ts | 1 + 13 files changed, 1061 insertions(+), 85 deletions(-) create mode 100644 tck/expected.ts create mode 100644 types/tck/expected.d.ts diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index fd54a21..8381a24 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -173,6 +173,45 @@ answer. If you read those fields yourself rather than through the runner, use `driverScope(module, env)` / `driverCapabilities(module, env)` from `open-ocpp-tck/driver` instead of narrowing the union by hand. +## Expected failures + +The scope table says what your CSMS **cannot drive**. This says what it drives +and **gets wrong**. Keeping them apart is the point: a known-red scenario keeps +its `DRIVABLE` row, still starts a container and still prints `FAIL` — what it +stops doing is ending the build. + +```ts +expectedFailures: { + "cert16-tc023-3-authorize-blocked": { + reason: "…the mechanism, cited, the way a scope row's is…", + finding: "…an upstream issue, or the row of your README's gap table…", + }, +}, +``` + +Same placement and same resolution rules as `scope`: on the module, read with +no credentials, and free to be a function of the environment +(`driverExpectedFailures(module, env)` if you read it yourself). + +**A declared scenario that passes fails the sweep**, as `UNEXPECTED PASS`, and +that half is not an inconvenience — it is what the mechanism is for. Without it +the list only ever grows, and an entry outlives the defect it documents. When +one fires, delete the entry or re-word it to say what is still true. + +Three rules, and they are the difference between a reviewed list and a mute: + +- `reason` names the mechanism. "Known red" is the observation being explained, + not the explanation. +- `finding` says where to go and read about it. `check-driver` rejects an empty + one, because a known-red nobody can look up is a claim nobody can review. +- **Never add an entry to quiet a flake.** There is no "expected flaky" status, + deliberately: a scenario that sometimes passes has a timing bug, and it will + be reported as an `UNEXPECTED PASS` the first time it does. Fix the scenario. + +`check-driver` also rejects an id that is stale, or that your scope table calls +`NOT_APPLICABLE` — a scenario that never starts can neither fail as declared nor +ever pass and delete the entry. + ### Two worked examples Two drivers ship here, and they are worth reading together because they solve @@ -188,8 +227,9 @@ For the mechanics of the contract — the operation switch, the record queries, the bootstrap verbs — either one works as a model. For **what to do when your CSMS cannot do something**, read `drivers/citrineos/`: it is the one with gaps, so it is the one that shows a scope table with `NOT_APPLICABLE` rows each citing -the endpoint that does not exist, a `records` object that omits `reservations` -outright rather than faking it, and a +the endpoint that does not exist, an `expectedFailures` list with a row that +stays `DRIVABLE` and red on purpose, a `records` object that omits +`reservations` outright rather than faking it, and a [README](drivers/citrineos/README.md) whose gap table names a source fact per row. diff --git a/README.md b/README.md index bc86e41..2e0cada 100644 --- a/README.md +++ b/README.md @@ -174,7 +174,34 @@ the check could not be evaluated" is a real outcome: because a driver answered with `unverifiable("")` instead of inventing a value. Exit code 0. -Only `FAIL` and `ERROR` exit non-zero. +`FAIL` and `ERROR` exit non-zero — unless your driver said so first. + +## Expected failures + +A CSMS you know gets one scenario wrong should not cost you the signal from the +other 46. Declare it, and the sweep keeps running it, keeps printing `FAIL`, +and stops failing the build for it: + +```ts +export const csmsDriver: CsmsDriverModule = { + expectedFailures: { + "cert16-tc023-3-authorize-blocked": { + reason: "…the mechanism, cited…", + finding: "…where the finding is written down…", + }, + }, +}; +``` + +The scope row stays `DRIVABLE`. Demoting it to `NOT_APPLICABLE` instead would +convert a finding about your CSMS into a silence about the harness, and +`check-driver` rejects an id that is declared both ways. + +**A declared scenario that passes fails the sweep**, reported as +`UNEXPECTED PASS`. That is the half that matters: it is how an entry gets +deleted the day upstream fixes the defect, instead of outliving it. There is +deliberately no "expected flaky" — a scenario that sometimes passes has a +timing bug, and the bug is the thing to fix. ## Environment diff --git a/VENDOR.md b/VENDOR.md index 81861f6..9169b28 100644 --- a/VENDOR.md +++ b/VENDOR.md @@ -57,12 +57,13 @@ strongest available demonstration that the core names no CSMS. | `tck/specs/index.ts` | `upstream-verbatim` | `scripts/steve-verify/runner/specs/index.ts` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `—` | | `tck/assert.ts` | `upstream-patched` | `src/cp/application/verification/assert.ts` | `2431f5f6c0df997d4d821d9af55689c1f0f2df199de1e9e4ed6f3fbaad4fc89e` | `76d0f293db4f6ecc5768affea0ca76d2841c147fa3687dbea1f9e950bfbd9298` | `patches/tck/assert.ts.patch` | | `tck/sim.ts` | `upstream-patched` | `scripts/steve-verify/runner/sim.ts` | `2bf2f78afe3434e7139cd62c3ff6d70f02defd39dd700611e7c5f7614260cd35` | `45f897f273ea73227bb9b30766127eb9ecae0a319ab9048cbc30428fafff7f32` | `patches/tck/sim.ts.patch` | -| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `cffb4876117c92b5273a5ef0aa6f58556c6136039c58a02c5aa6adcf156c1206` | `patches/tck/main.ts.patch` | +| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `457981bdfd6bb42253acc62fd76ca49043bd2e4e47c33476a4415a1dd7414040` | `patches/tck/main.ts.patch` | | `tck/driver.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/index.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/driver-registry.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/capabilities.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/scope.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | +| `tck/expected.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/time.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/unverifiable.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/wait.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | diff --git a/package.json b/package.json index 4b394f4..03f9c01 100644 --- a/package.json +++ b/package.json @@ -52,6 +52,10 @@ "types": "./types/tck/scope.d.ts", "default": "./tck/scope.ts" }, + "./expected": { + "types": "./types/tck/expected.d.ts", + "default": "./tck/expected.ts" + }, "./unverifiable": { "types": "./types/tck/unverifiable.d.ts", "default": "./tck/unverifiable.ts" diff --git a/patches/tck/main.ts.patch b/patches/tck/main.ts.patch index 892ba05..6736b32 100644 --- a/patches/tck/main.ts.patch +++ b/patches/tck/main.ts.patch @@ -1,6 +1,6 @@ --- /tmp/ocpp-upstream/scripts/steve-verify/runner/main.ts 2026-07-31 20:05:48 +++ tck/main.ts 2026-07-31 20:12:10 -@@ -1,39 +1,65 @@ +@@ -1,39 +1,81 @@ #!/usr/bin/env bun +// Derived from shiv3/ocpp-cp-simulator scripts/steve-verify/runner/main.ts @ 604054adb0d7d7129a26a5f1ad2d5fdc290d1ca1 (Apache-2.0). Modified: the STEVE_DRIVER=api|ui selection is replaced by a CSMS driver loaded through ./driver-registry; a per-driver scope table (./scope) is consulted BEFORE any container starts and yields the NOT APPLICABLE verdict; UnsupportedOperationError (./driver) thrown out of drive() degrades to NOT APPLICABLE with a stderr WARNING; the PARTIAL verdict and the `skipped` summary column were added; the exit code is non-zero only for FAIL/ERROR; parallel lanes derive from the resolved station list instead of the fixed CERTCP1..3 trio; the SteVe capability probe is dropped. + @@ -40,7 +40,16 @@ + * - PARTIAL -- zero FAILs but at least one check degraded to SKIPPED + * because the driver answered with assert.ts's UNVERIFIABLE sentinel. + * Exit code 0: a check that could not be evaluated is not a defect. -+ * Only FAIL and ERROR make the process exit non-zero. ++ * ++ * The verdict is what the CSMS did; the EXIT CODE is what that means for this ++ * driver, and the two stopped being the same thing when expected.ts arrived. ++ * A FAIL the driver declared expected exits 0 -- it is a finding already ++ * written down, not news -- and a PASS it declared expected exits 1, because ++ * a list that cannot shrink is a mute. So: ++ * ++ * exit non-zero <= an undeclared FAIL/ERROR, or a declared one that passed. ++ * ++ * A driver that declares nothing keeps the original rule exactly. */ import { mkdirSync } from "node:fs"; @@ -52,11 +61,18 @@ +import { + CSMS_OPERATION_ACTIONS, + driverCapabilities, ++ driverExpectedFailures, + driverScope, + UnsupportedOperationError, +} from "./driver"; +import { loadDriverModule } from "./driver-registry"; +import { ++ expectedFailureCoverage, ++ expectedFailureFor, ++ type ExpectedFailureEntry, ++ type ExpectedFailureTable, ++} from "./expected"; ++import { + scopeCoverage, + templateIdsWithStatus, + type ScopeStatus, @@ -92,7 +108,7 @@ import { AUTHLIST_RESERVATION_SPECS, AUTHORIZE_SPECS, -@@ -43,13 +69,25 @@ +@@ -43,13 +85,25 @@ } from "./specs/index"; import type { ScenarioSpec } from "./spec-types"; import { sleep } from "./util"; @@ -123,7 +139,7 @@ // --------------------------------------------------------------------------- // Runner core // --------------------------------------------------------------------------- -@@ -60,95 +98,212 @@ +@@ -60,95 +114,242 @@ timeoutSecs?: number; } @@ -177,12 +193,6 @@ + return (await scopeTable())?.[templateId]; } -+/** Result of one scenario execution -- either it produced checks, or the -+ * driver declared the scenario undrivable partway through drive(). */ -+type ScenarioRun = -+ | { kind: "checked"; rec: AssertRecorder } -+ | { kind: "not-applicable"; reason: string }; -+ /** - * Issue #184 Task 3: picks the SteveTx (transaction/reservation-assertion) - * driver for this run, using the SAME `STEVE_DRIVER` flag as @@ -193,6 +203,34 @@ - * `latestReservationPk`/`reservationStatus` to a DB fallback internally - * (no REST reservations endpoint exists; see steve.ts's `SteveTx` doc - * comment). ++ * The driver's expected-failure entry for one scenario, or undefined when it ++ * declared none -- in which case every failure is a failure, which is what a ++ * driver with nothing to declare should keep saying. ++ * ++ * Offline for the same reason {@link scopeEntryFor} is, and it matters here ++ * too: `ocpp-tck run` must be able to tell you a red scenario was expected ++ * without first demanding the credential it is not going to use. + */ +-function createDb(env: NodeJS.ProcessEnv = process.env): SteveTx { +- const driver = (env.STEVE_DRIVER ?? "api").toLowerCase(); +- const dbCfg = defaultSteveConfig(env); +- if (driver === "ui") { +- return new SteveDb(dbCfg); +- } +- return new SteveApiDb(defaultSteveApiDbConfig(env), dbCfg); ++async function expectedFailureEntryFor( ++ templateId: string, ++): Promise { ++ return expectedFailureFor(await expectedFailureTable(), templateId); ++} ++ ++/** Result of one scenario execution -- either it produced checks, or the ++ * driver declared the scenario undrivable partway through drive(). */ ++type ScenarioRun = ++ | { kind: "checked"; rec: AssertRecorder } ++ | { kind: "not-applicable"; reason: string }; ++ ++/** + * The driver, loaded once per process and shared by every lane. Replaces + * upstream's `STEVE_DRIVER=api|ui` fork: which CSMS is under test is a + * `CSMS_DRIVER` module specifier, and specs never learn which. @@ -203,17 +241,13 @@ + * a scenario this CSMS cannot drive is reported NOT APPLICABLE without the + * driver ever needing valid credentials -- which is what makes the scope table + * reviewable offline. That only holds because the preflight stops at stage one. - */ --function createDb(env: NodeJS.ProcessEnv = process.env): SteveTx { -- const driver = (env.STEVE_DRIVER ?? "api").toLowerCase(); -- const dbCfg = defaultSteveConfig(env); -- if (driver === "ui") { -- return new SteveDb(dbCfg); -- } -- return new SteveApiDb(defaultSteveApiDbConfig(env), dbCfg); ++ */ +let driverModulePromise: Promise | undefined; +let driverPartsPromise: Promise | undefined; +let scopeTablePromise: Promise | undefined; ++let expectedFailurePromise: ++ | Promise ++ | undefined; + +/** + * The one environment this process shows a driver, named once so that the @@ -246,6 +280,18 @@ + return scopeTablePromise; +} + ++/** The expected-failure list, resolved once with the SAME `ENV` as the scope ++ * table and as `create()`. A list describing one release line while the ++ * requests target the other excuses the wrong scenarios. */ ++async function expectedFailureTable(): Promise< ++ ExpectedFailureTable | undefined ++> { ++ expectedFailurePromise ??= driverModule().then((module) => ++ driverExpectedFailures(module, ENV), ++ ); ++ return expectedFailurePromise; ++} ++ +/** + * Fills in the optional halves of CsmsRecords with stubs that throw + * UnsupportedOperationError, carrying the driver's own reason. @@ -397,7 +443,7 @@ 30_000, ); } catch (err) { -@@ -176,13 +331,25 @@ +@@ -176,13 +377,25 @@ if (spec.drive) { process.stderr.write(`[runner] running drive() for ${spec.templateId}\n`); @@ -430,7 +476,7 @@ } else { process.stderr.write( `[runner] no drive() defined (CP-only scenario) -- nothing to do while it runs\n`, -@@ -190,15 +357,18 @@ +@@ -190,15 +403,18 @@ driveState = undefined as D; } @@ -452,7 +498,7 @@ // it a FAIL in a swept scenario is un-post-mortem-able: the sim container // is already stopped+removed by the time assert() reports, so this // capture is the only surviving record of what was (or wasn't) on the -@@ -218,6 +388,10 @@ +@@ -218,6 +434,10 @@ ); } @@ -463,7 +509,7 @@ const frames = parseLog(lines.join("\n")); const rec = new AssertRecorder(); -@@ -228,34 +402,53 @@ +@@ -228,34 +448,53 @@ frames, lines, rec, @@ -526,7 +572,7 @@ // --------------------------------------------------------------------------- // eslint-disable-next-line @typescript-eslint/no-explicit-any -@@ -265,8 +458,6 @@ +@@ -265,8 +504,6 @@ "remotetrigger-smartcharging": REMOTETRIGGER_SMARTCHARGING_SPECS, firmware: FIRMWARE_SPECS, authorize: AUTHORIZE_SPECS, @@ -535,7 +581,7 @@ all: [ ...CORE_SPECS, ...AUTHLIST_RESERVATION_SPECS, -@@ -286,66 +477,109 @@ +@@ -286,66 +523,191 @@ .map((spec) => [spec.templateId, spec]), ); @@ -620,6 +666,10 @@ - errorMessage?: string; - }; + isolatedRetry?: RetryOutcome; ++ /** The driver's declaration that this scenario is known to fail, attached ++ * whether or not it did. Both directions are reported: a declared FAIL is ++ * excused, a declared PASS fails the sweep. */ ++ expected?: ExpectedFailureEntry; } /** @@ -629,17 +679,92 @@ - * sweep -- bash's per-scenario isolation comes for free from each - * run-scenario.sh invocation being a separate subprocess; this is the - * same-process equivalent. ++ * Did this scenario really fail -- after the isolated retry has had its say? ++ * ++ * ONE definition, deliberately, because there are now two consumers: the exit ++ * code, and the comparison against the driver's expected-failure list. Written ++ * twice they would drift, and the drift would be silent in the worst possible ++ * place -- a row excused by one rule and counted by the other. ++ * ++ * A parallel-lane FAIL/ERROR that did not fail its isolated retry is a flake, ++ * not a failure; see {@link retryFailedOutcomesIsolated}. + */ ++function effectivelyFailed(outcome: ScenarioOutcome): boolean { ++ if (!isFailure(outcome.verdict)) return false; ++ return ( ++ outcome.isolatedRetry === undefined || ++ isFailure(outcome.isolatedRetry.verdict) ++ ); ++} ++ ++/** ++ * How an outcome lands against what the driver declared. This -- not the ++ * verdict -- is what the exit code reads. ++ * ++ * `expected-fail` is the whole point of the list: a finding already written ++ * down is not news, and muting it one scenario at a time is what lets the ++ * other 46 be reported at all. ++ * ++ * `unexpected-pass` is what keeps the list from becoming the job-level mute it ++ * replaced. It fires on a declared scenario that did NOT effectively fail, ++ * which includes the case where it failed in parallel and passed its isolated ++ * retry -- there is no "expected flaky", so an entry that passes any way at ++ * all is an entry to delete or to re-word. ++ */ ++type SweepStanding = ++ | "ok" ++ | "expected-fail" ++ | "unexpected-fail" ++ | "unexpected-pass"; ++ ++function standingOf(outcome: ScenarioOutcome): SweepStanding { ++ const failed = effectivelyFailed(outcome); ++ if (outcome.expected === undefined) { ++ return failed ? "unexpected-fail" : "ok"; ++ } ++ if (failed) return "expected-fail"; ++ // NOT APPLICABLE is unreachable here: check-driver rejects an id that is ++ // both declared expected-failing and NOT_APPLICABLE in the scope table, and ++ // the runtime UnsupportedOperationError escape would be the same ++ // contradiction arriving late. Reported as an unexpected pass rather than ++ // silently tolerated -- "it never ran" is not "it was fixed", and the ++ // message below says which. ++ return "unexpected-pass"; ++} ++ ++/** Why an unexpected pass is one, in the words a reader needs to tell a fix ++ * from a flake from a contradiction without re-reading this file. */ ++function unexpectedPassDetail(outcome: ScenarioOutcome): string { ++ if (outcome.verdict === "NOT APPLICABLE") { ++ return "it never ran (NOT APPLICABLE), so it cannot be failing as declared"; ++ } ++ if (isFailure(outcome.verdict)) { ++ return ( ++ `it failed in the parallel lane and ${outcome.isolatedRetry?.verdict} ` + ++ "on its isolated retry -- either upstream fixed it, or the scenario is " + ++ "flaky and the flake is the bug to fix" ++ ); ++ } ++ return `it came back ${outcome.verdict} outright -- the finding looks fixed`; ++} ++ ++/** + * Runs one scenario for a group sweep, isolating a thrown exception (e.g. a + * bounded wait's fail-hard timeout rejection propagating out of drive()) to + * THIS scenario's outcome instead of aborting the whole sweep. + * + * The scope table is consulted FIRST -- a NOT_APPLICABLE scenario never + * starts a container. - */ ++ */ async function runOneForSweep( spec: ScenarioSpec, cpId: string, ): Promise { ++ // Read before the run and carried on every branch, including the ones that ++ // never start a container: a declaration that only got attached to failures ++ // could never report the case the list exists to catch, which is the row ++ // that stopped failing. ++ const expected = await expectedFailureEntryFor(spec.templateId); + const scope = await scopeEntryFor(spec.templateId); + if (scope?.status === "NOT_APPLICABLE") { + process.stderr.write( @@ -653,6 +778,7 @@ + failed: null, + skipped: null, + reason: scope.reason, ++ expected, + }; + } + @@ -668,6 +794,7 @@ + failed: null, + skipped: null, + reason: run.reason, ++ expected, + }; + } return { @@ -680,15 +807,17 @@ + checks: run.rec.total, + failed: run.rec.failed, + skipped: run.rec.skipped, ++ expected, }; } catch (err) { const message = -@@ -359,58 +593,40 @@ +@@ -359,58 +721,47 @@ verdict: "ERROR", checks: null, failed: null, + skipped: null, errorMessage: message, ++ expected, }; } } @@ -737,6 +866,12 @@ - * 3 single scenarios + 1 UI-fallback single scenario, not a full sweep) -- - * left for Task 4's full-verification pass to confirm or refute. + * PARTIAL and NOT APPLICABLE are never retried: neither is a failure. ++ * ++ * A DECLARED expected failure IS retried, and that is not waste. The retry is ++ * what turns "it failed" into "it failed with no lane contending", which is ++ * the only evidence that would let the declaration be deleted; a declared row ++ * that passes its isolated retry is reported as an unexpected pass, because ++ * there is no "expected flaky". */ async function retryFailedOutcomesIsolated( outcomes: ScenarioOutcome[], @@ -757,7 +892,7 @@ ); for (const outcome of toRetry) { -@@ -432,13 +648,14 @@ +@@ -432,13 +783,14 @@ verdict: retryOutcome.verdict, checks: retryOutcome.checks, failed: retryOutcome.failed, @@ -774,7 +909,7 @@ : " -- CONFIRMED (fails isolated too, not a parallel-lane artifact)\n"), ); } -@@ -451,11 +668,43 @@ +@@ -451,11 +803,43 @@ return new Date().toISOString().replace(/\.\d{3}Z$/, "Z"); } @@ -823,13 +958,25 @@ async function writeSummary( groupName: string, outcomes: ScenarioOutcome[], -@@ -465,45 +714,55 @@ +@@ -465,45 +849,97 @@ const rows = outcomes.map((o) => { const checks = o.checks === null ? "-" : String(o.checks); const failed = o.failed === null ? "-" : String(o.failed); - const base = `| ${o.templateId} | ${o.cpId} | ${o.verdict} | ${checks} | ${failed} |`; + const skipped = o.skipped === null ? "-" : String(o.skipped); -+ const verdict = o.reason ? `${o.verdict} (${o.reason})` : o.verdict; ++ let verdict = o.reason ? `${o.verdict} (${o.reason})` : o.verdict; ++ // The declaration is rendered INTO the verdict cell rather than into a ++ // column of its own, so that a reader scanning for red sees, in the same ++ // glance, whether it was already known -- and, for an unexpected pass, ++ // sees the row that has to be deleted. ++ const standing = standingOf(o); ++ if (standing === "expected-fail") { ++ verdict = `${verdict} — EXPECTED FAIL: ${o.expected?.reason} (${o.expected?.finding})`; ++ } else if (standing === "unexpected-pass") { ++ verdict = ++ `${verdict} — UNEXPECTED PASS: declared expected-failing, but ` + ++ `${unexpectedPassDetail(o)}. Delete the entry or re-word it.`; ++ } + const base = `| ${o.templateId} | ${o.cpId} | ${verdict} | ${checks} | ${failed} | ${skipped} |`; if (!anyRetried) return base; if (!o.isolatedRetry) return `${base} - |`; @@ -861,6 +1008,36 @@ + `${partialCount} PARTIAL (checks the driver could not evaluate were SKIPPED), ` + + `${naCount} NOT APPLICABLE (out of scope for this CSMS). Neither fails the sweep.`, + ); ++ } ++ const expectedFails = outcomes.filter( ++ (o) => standingOf(o) === "expected-fail", ++ ); ++ const unexpectedPasses = outcomes.filter( ++ (o) => standingOf(o) === "unexpected-pass", ++ ); ++ if (expectedFails.length > 0) { ++ notes.push( ++ "", ++ `${expectedFails.length} EXPECTED FAIL — declared by this driver as a ` + ++ "known finding about the CSMS, so the sweep still exits 0. The scope " + ++ "rows stay DRIVABLE and the scenarios still ran; only the exit code " + ++ "changed. Each entry names where the finding is written down:", ++ ...expectedFails.map( ++ (o) => `- \`${o.templateId}\` — ${o.expected?.finding}`, ++ ), ++ ); ++ } ++ if (unexpectedPasses.length > 0) { ++ notes.push( ++ "", ++ `${unexpectedPasses.length} UNEXPECTED PASS — declared expected-failing ` + ++ "and did not fail. **This fails the sweep**, on purpose: it is how a " + ++ "known-red entry gets deleted when it stops being true, instead of " + ++ "outliving the defect it documents.", ++ ...unexpectedPasses.map( ++ (o) => `- \`${o.templateId}\` — ${unexpectedPassDetail(o)}`, ++ ), ++ ); + } if (anyRetried) { const flakeCount = outcomes.filter( @@ -894,7 +1071,7 @@ "", header, separator, -@@ -518,54 +777,53 @@ +@@ -518,54 +954,53 @@ } /** Sequential or parallel group sweep. Writes results/summary.md and exits @@ -974,7 +1151,7 @@ ); } else { for (let i = 0; i < specs.length; i++) { -@@ -574,11 +832,11 @@ +@@ -574,11 +1009,11 @@ } if (retryFailedIsolated) { @@ -988,7 +1165,7 @@ "(a sequential sweep is already isolated).\n", ); } -@@ -588,45 +846,51 @@ +@@ -588,45 +1023,85 @@ for (const o of outcomes) { const retrySuffix = o.isolatedRetry ? ` [isolated retry: ${o.isolatedRetry.verdict}${ @@ -997,9 +1174,16 @@ }]` : ""; + const reasonSuffix = o.reason ? ` -- ${o.reason}` : ""; ++ const standing = standingOf(o); ++ const declaredSuffix = ++ standing === "expected-fail" ++ ? ` [EXPECTED FAIL: ${o.expected?.reason} (${o.expected?.finding})]` ++ : standing === "unexpected-pass" ++ ? ` [UNEXPECTED PASS: declared expected-failing, but ${unexpectedPassDetail(o)}]` ++ : ""; process.stderr.write( - ` ${o.verdict}: ${o.templateId} (${o.cpId}, ${o.checks ?? "-"} checks, ${o.failed ?? "-"} failed)${retrySuffix}\n`, -+ ` ${o.verdict}: ${o.templateId} (${o.cpId}, ${o.checks ?? "-"} checks, ${o.failed ?? "-"} failed, ${o.skipped ?? "-"} skipped)${retrySuffix}${reasonSuffix}\n`, ++ ` ${o.verdict}: ${o.templateId} (${o.cpId}, ${o.checks ?? "-"} checks, ${o.failed ?? "-"} failed, ${o.skipped ?? "-"} skipped)${retrySuffix}${declaredSuffix}${reasonSuffix}\n`, ); } @@ -1011,13 +1195,18 @@ - // sweep. Anything else non-PASS (no retry attempted, or still non-PASS - // isolated) counts as a real failure. + // A parallel-lane FAIL/ERROR that did not fail on its isolated retry is a -+ // flake, not a real failure -- it does not fail the sweep. ++ // flake, not a real failure -- it does not fail the sweep. Everything below ++ // reads that through effectivelyFailed(), via standingOf(). const badOutcomes = outcomes.filter( - (o) => o.verdict !== "PASS" && o.isolatedRetry?.verdict !== "PASS", -+ (o) => -+ isFailure(o.verdict) && -+ (o.isolatedRetry === undefined || isFailure(o.isolatedRetry.verdict)), ++ (o) => standingOf(o) === "unexpected-fail", ); ++ const expectedFails = outcomes.filter( ++ (o) => standingOf(o) === "expected-fail", ++ ); ++ const unexpectedPasses = outcomes.filter( ++ (o) => standingOf(o) === "unexpected-pass", ++ ); const flakeCount = outcomes.filter( - (o) => o.verdict !== "PASS" && o.isolatedRetry?.verdict === "PASS", + (o) => @@ -1031,13 +1220,32 @@ + `[runner] ${flakeCount} parallel-only flake(s) in group '${groupName}' (did not fail on isolated retry) -- see ${summaryPath}\n`, ); } ++ if (expectedFails.length > 0) { ++ process.stderr.write( ++ `[runner] ${expectedFails.length} scenario(s) in group '${groupName}' failed as this driver declared they would -- not a build failure; see ${summaryPath}\n`, ++ ); ++ } ++ // Reported BEFORE the ordinary failures and counted separately, because the ++ // two need opposite fixes: an unexpected failure is a defect to chase, an ++ // unexpected pass is a line to delete. ++ if (unexpectedPasses.length > 0) { ++ process.stderr.write( ++ `[runner] ${unexpectedPasses.length} scenario(s) in group '${groupName}' are declared expected-failing and did NOT fail:\n`, ++ ); ++ for (const o of unexpectedPasses) { ++ process.stderr.write( ++ ` ${o.templateId}: ${unexpectedPassDetail(o)} -- delete its expected-failure entry, or re-word it to say what is still true.\n`, ++ ); ++ } ++ } if (badOutcomes.length > 0) { process.stderr.write( `[runner] ${badOutcomes.length}/${outcomes.length} scenario(s) in group '${groupName}' FAILed or errored -- see ${summaryPath}\n`, ); - process.exit(1); -+ return 1; } ++ if (badOutcomes.length > 0 || unexpectedPasses.length > 0) return 1; ++ + const partialCount = outcomes.filter((o) => o.verdict === "PARTIAL").length; + const naCount = outcomes.filter((o) => o.verdict === "NOT APPLICABLE").length; process.stderr.write( @@ -1045,8 +1253,11 @@ - (flakeCount > 0 - ? ` (${flakeCount} flake(s) resolved by isolated retry).\n` - : ".\n"), -+ `[runner] no failures in group '${groupName}': ${outcomes.length} scenario(s), ` + ++ `[runner] no unexpected failures in group '${groupName}': ${outcomes.length} scenario(s), ` + + `${partialCount} PARTIAL, ${naCount} NOT APPLICABLE` + ++ (expectedFails.length > 0 ++ ? `, ${expectedFails.length} EXPECTED FAIL` ++ : "") + + (flakeCount > 0 ? `, ${flakeCount} flake(s) resolved by isolated retry` : "") + + ".\n", ); @@ -1055,7 +1266,7 @@ } // --------------------------------------------------------------------------- -@@ -642,6 +906,8 @@ +@@ -642,6 +1117,8 @@ cpId: string; connector?: number; timeoutSecs?: number; @@ -1064,7 +1275,7 @@ } function requireValue(argv: string[], index: number, flag: string): string { -@@ -663,37 +929,80 @@ +@@ -663,37 +1140,80 @@ return n; } @@ -1162,7 +1373,7 @@ if (argv[0] === "run-all") { group = "all"; -@@ -708,18 +1017,27 @@ +@@ -708,18 +1228,27 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1193,7 +1404,7 @@ process.exit(1); } -@@ -749,9 +1067,11 @@ +@@ -749,9 +1278,11 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1206,7 +1417,7 @@ process.exit(1); } } -@@ -764,35 +1084,431 @@ +@@ -764,35 +1295,486 @@ cpId, connector, timeoutSecs, @@ -1298,10 +1509,12 @@ + let module: CsmsDriverModule; + let scope: ScopeTable | undefined; + let capabilities: CsmsCapabilities | undefined; ++ let expected: ExpectedFailureTable | undefined; + try { + module = await driverModule(); + scope = await scopeTable(); + capabilities = driverCapabilities(module, ENV); ++ expected = await expectedFailureTable(); + } catch (err) { + process.stderr.write( + `${err instanceof Error ? err.message : String(err)}\n`, @@ -1364,7 +1577,7 @@ + "invisible to every consumer.", + ); + } - ++ + const reasonless = Object.entries(scope) + .filter(([, entry]) => !entry.reason?.trim()) + .map(([id]) => id); @@ -1376,6 +1589,46 @@ + ); + } + } + ++ // Everything about the expected-failure list that is knowable offline. What ++ // is NOT knowable here is whether those scenarios still fail -- only a sweep ++ // answers that, by reporting an UNEXPECTED PASS. ++ if (expected) { ++ const drift = expectedFailureCoverage(expected, scope, registered); ++ if (drift.stale.length > 0) { ++ problems.push( ++ `${drift.stale.length} expected-failure entry(ies) name a scenario ` + ++ `nobody registers: ${drift.stale.join(", ")}.\n` + ++ " -> delete the entry, or fix the templateId if the scenario was " + ++ "renamed. An entry under an old id excuses nothing and can never be " + ++ "cleaned up by a passing run.", ++ ); ++ } ++ if (drift.notApplicable.length > 0) { ++ problems.push( ++ `${drift.notApplicable.length} expected-failure entry(ies) are also ` + ++ `NOT_APPLICABLE in the scope table: ${drift.notApplicable.join(", ")}.\n` + ++ " -> both cannot be true. A NOT_APPLICABLE scenario never starts, " + ++ "so it can neither fail as declared nor ever pass and delete the entry.", ++ ); ++ } ++ if (drift.reasonless.length > 0) { ++ problems.push( ++ `${drift.reasonless.length} expected-failure entry(ies) carry an ` + ++ `empty reason: ${drift.reasonless.join(", ")}. "Known red" is the ` + ++ "observation, not the mechanism.", ++ ); ++ } ++ if (drift.findingless.length > 0) { ++ problems.push( ++ `${drift.findingless.length} expected-failure entry(ies) carry an ` + ++ `empty finding: ${drift.findingless.join(", ")}.\n` + ++ " -> name where it is written down: an upstream issue, or the row " + ++ "of this driver's README gap table that carries the evidence. A " + ++ "known-red with nowhere to read about it is how the list rots.", ++ ); ++ } ++ } + + if (capabilities) { + const known = new Set(CSMS_OPERATION_ACTIONS); @@ -1411,6 +1664,7 @@ + NOT_APPLICABLE: templateIdsWithStatus(scope, "NOT_APPLICABLE").length, + } + : null, ++ expectedFailures: expected ? Object.keys(expected).sort() : null, + problems, + warnings, + }; @@ -1440,6 +1694,18 @@ + : "") + + ".\n", + ); ++ // Listed rather than counted: this is the one declaration whose whole job is ++ // to shrink, and a number does not tell a reviewer which rows to go and check. ++ if (summary.expectedFailures && summary.expectedFailures.length > 0) { ++ process.stderr.write( ++ ` ${summary.expectedFailures.length} scenario(s) declared expected-failing ` + ++ "-- each still runs, still prints FAIL, and fails the sweep the day it " + ++ "passes:\n", ++ ); ++ for (const id of summary.expectedFailures) { ++ process.stderr.write(` ${id}: ${expected?.[id]?.finding}\n`); ++ } ++ } + return 0; +} + @@ -1654,7 +1920,7 @@ } const spec = SPECS_BY_TEMPLATE_ID.get(args.templateId); -@@ -800,18 +1516,29 @@ +@@ -800,18 +1782,70 @@ process.stderr.write( `Unknown template id: ${args.templateId} (known: ${[...SPECS_BY_TEMPLATE_ID.keys()].join(", ")})\n`, ); @@ -1665,17 +1931,26 @@ - const rec = await runScenario(spec, options); - process.exit(rec.verdict === "PASS" ? 0 : 1); -} ++ // The same declaration the sweep reads, so that running one scenario by ++ // hand means what running it in the sweep means. Without this, ++ // `bun run e2e:smoke` -- which names an expected-failing scenario ++ // explicitly -- would contradict `bun run e2e` on the same commit. ++ const expected = await expectedFailureEntryFor(spec.templateId); + +-if (import.meta.main) { +- main().catch((err) => { + // Scope table first -- a NOT_APPLICABLE scenario never starts a container. + const scope = await scopeEntryFor(spec.templateId); + if (scope?.status === "NOT_APPLICABLE") { -+ process.stderr.write( + process.stderr.write( +- `Fatal: ${err instanceof Error ? (err.stack ?? err.message) : String(err)}\n`, + `[runner] RESULT: ${spec.templateId} NOT APPLICABLE (no container started): ${scope.reason}\n`, -+ ); -+ return 0; + ); +- process.exit(1); +- }); ++ return unexpectedPassExit(spec.templateId, "NOT APPLICABLE", expected); + } - --if (import.meta.main) { -- main().catch((err) => { ++ + const options: RunOptions = { + cpId: args.cpId, + connector: args.connector, @@ -1683,13 +1958,45 @@ + }; + const run = await runScenario(spec, options); + if (run.kind === "not-applicable") { - process.stderr.write( -- `Fatal: ${err instanceof Error ? (err.stack ?? err.message) : String(err)}\n`, ++ process.stderr.write( + `[runner] RESULT: ${spec.templateId} NOT APPLICABLE: ${run.reason}\n`, - ); -- process.exit(1); -- }); -+ return 0; ++ ); ++ return unexpectedPassExit(spec.templateId, "NOT APPLICABLE", expected); ++ } ++ ++ const verdict = verdictForRecorder(run.rec); ++ if (isFailure(verdict)) { ++ if (expected) { ++ process.stderr.write( ++ `[runner] ${spec.templateId} EXPECTED FAIL: ${expected.reason} ` + ++ `(${expected.finding}) -- declared by this driver, exit 0.\n`, ++ ); ++ return 0; ++ } ++ return 1; + } -+ return isFailure(verdictForRecorder(run.rec)) ? 1 : 0; ++ return unexpectedPassExit(spec.templateId, verdict, expected); } ++ ++/** ++ * Exit code for a single scenario that did NOT fail: 0 normally, 1 when the ++ * driver declared it expected-failing. ++ * ++ * A single run has no isolated retry, so there is no flake to distinguish ++ * here -- one non-failure IS the whole evidence, and the message says so ++ * rather than asserting the finding is fixed. ++ */ ++function unexpectedPassExit( ++ templateId: string, ++ verdict: Verdict, ++ expected: ExpectedFailureEntry | undefined, ++): number { ++ if (!expected) return 0; ++ process.stderr.write( ++ `[runner] ${templateId} UNEXPECTED PASS: this driver declares it ` + ++ `expected-failing (${expected.reason} -- ${expected.finding}), and it ` + ++ `came back ${verdict}. Delete the expected-failure entry, or re-word it ` + ++ "to say what is still true.\n", ++ ); ++ return 1; ++} diff --git a/tck/driver.ts b/tck/driver.ts index 0b65c6c..b3cb9c3 100644 --- a/tck/driver.ts +++ b/tck/driver.ts @@ -54,6 +54,7 @@ * describing a capability gap; declare it in the driver's scope table. */ +import type { ExpectedFailureTable } from "./expected"; import type { ScopeTable } from "./scope"; // --------------------------------------------------------------------------- @@ -539,6 +540,24 @@ export interface CsmsDriverModule { * function of the environment. Resolve it with {@link driverCapabilities}. */ readonly capabilities?: EnvDependent; + /** + * Scenarios this CSMS is KNOWN to fail -- drivable, run, red, and understood. + * + * The complement of {@link CsmsDriverModule.scope}, not a part of it: a + * listed scenario keeps its DRIVABLE row, still starts a container and still + * prints FAIL. What the list changes is only the sweep's exit code, so that + * a job muted for one finding can still report every other scenario. An + * entry that PASSES fails the sweep in the other direction -- see + * {@link ./expected}. + * + * Absent = "every failure is a failure", which is what a driver with nothing + * to declare should keep saying. Free to be a function of the environment + * for the same reason `scope` is: a CSMS with two release lines does not + * have the same defects on both. Resolve it with + * {@link driverExpectedFailures}. + */ + readonly expectedFailures?: EnvDependent; + /** Called once per process, and the result is shared by every parallel lane, * so what it returns must be safe to use concurrently and must hold no * per-lane state. The contract used to promise one instance per lane, which @@ -583,3 +602,16 @@ export function driverCapabilities( ? capabilities(env) : capabilities; } + +/** + * {@link driverScope} for the expected-failure list, and it must be resolved + * with the SAME env: a list naming defects of one release line, applied to a + * sweep pointed at the other, excuses the wrong scenarios in both directions. + */ +export function driverExpectedFailures( + module: CsmsDriverModule, + env: CsmsEnv, +): ExpectedFailureTable | undefined { + const expected = module.expectedFailures; + return typeof expected === "function" ? expected(env) : expected; +} diff --git a/tck/expected.ts b/tck/expected.ts new file mode 100644 index 0000000..5f5c604 --- /dev/null +++ b/tck/expected.ts @@ -0,0 +1,107 @@ +// Copyright 2026 Julien Herr +// SPDX-License-Identifier: Apache-2.0 +/** + * expected.ts -- the TYPE of a driver's expected-failure list, and the drift + * checks that keep it honest. The DATA lives in each driver + * (drivers//expected.ts): which scenarios a CSMS gets wrong is a fact about + * that CSMS, not about this harness. + * + * WHY THIS IS NOT A SCOPE STATUS. {@link ./scope} answers "can this CSMS drive + * the scenario at all?", and its rules forbid demoting a row to + * NOT_APPLICABLE to make a red scenario go away -- that converts a finding + * about the CSMS into a silence about the harness. This file answers the other + * question: the scenario IS drivable, it runs, it prints FAIL, and we already + * know why. The row stays DRIVABLE; only the exit code changes. + * + * WHAT IT REPLACES. Before it, the only vocabulary for "we know this one is + * red" was `continue-on-error` on the whole CI job -- job-granular where the + * intent is scenario-granular. A broken compose file, a failed provisioning + * step and a regression from one failure to twenty were all as green as the + * finding the job was muted for, so the job could not report the one thing it + * existed to report: every OTHER scenario. + * + * THE LIST HAS TO BE ABLE TO SHRINK, or it is just a quieter mute. So an + * expected failure that PASSES is itself a failure -- reported as + * UNEXPECTED PASS, with a non-zero exit -- which is how a row gets deleted + * when upstream fixes the defect, instead of outliving it. + * + * RULES FOR A DRIVER AUTHOR EDITING ITS LIST + * - `reason` names the mechanism, cited, the way a scope row's does. "Fails + * on this CSMS" is not a reason; it is the observation being explained. + * - `finding` says where the finding is written down -- an upstream issue, or + * the row of the driver README's gap table that carries the evidence. A + * known-red with nowhere to read about it is a claim nobody can review, and + * it is the shape this list rots into if the field is optional. + * - NEVER add a row to quiet a flake. A scenario that sometimes passes has a + * timing bug, and the fix is the scenario. There is deliberately no + * "expected flaky" status here: it would re-create the job-level mute one + * scenario at a time. + */ +import type { ScopeTable } from "./scope"; + +export interface ExpectedFailureEntry { + /** The mechanism, cited. Not "known red" -- WHY it is red. */ + reason: string; + /** + * Where the finding is recorded: an upstream issue URL, or the driver + * README's gap-table row. Required, and checked non-empty by + * `ocpp-tck check-driver`. + */ + finding: string; +} + +/** One entry per scenario `templateId` this CSMS is known to fail. */ +export type ExpectedFailureTable = Readonly< + Record +>; + +export function expectedFailureFor( + table: ExpectedFailureTable | undefined, + templateId: string, +): ExpectedFailureEntry | undefined { + return table?.[templateId]; +} + +/** + * Everything about a list that can be established WITHOUT running anything. + * + * Four kinds, and each is a way the list stops meaning what it says: + * + * `stale` -- an id nobody registers. Usually a rename, and it silently stops + * excusing anything; the scenario it was written for now fails the build under + * its new name, which at least is loud. The reverse of the same rename is not: + * a row kept under the OLD id while the scenario is fixed under the new one + * never gets the UNEXPECTED PASS that would delete it. + * + * `notApplicable` -- the scope table says the scenario never runs on this + * CSMS, and this list says it runs and fails. Both cannot be true. Left + * unchecked it is the quiet way a driver ends up with a row that can never + * fire and can never be cleaned up, because the only thing that deletes a row + * is the scenario passing, and a scenario that never starts never passes. + * + * `reasonless` / `findingless` -- see the rules in this file's header. + * + * What it deliberately CANNOT check is whether the scenario actually fails. + * Only a sweep knows that, and it is the sweep that reports UNEXPECTED PASS. + */ +export function expectedFailureCoverage( + table: ExpectedFailureTable, + scope: ScopeTable | undefined, + registeredTemplateIds: readonly string[], +): { + stale: string[]; + notApplicable: string[]; + reasonless: string[]; + findingless: string[]; +} { + const registered = new Set(registeredTemplateIds); + const ids = Object.keys(table).sort(); + return { + stale: ids.filter((id) => !registered.has(id)), + notApplicable: ids.filter( + (id) => scope?.[id]?.status === "NOT_APPLICABLE", + ), + reasonless: ids.filter((id) => !table[id]?.reason?.trim()), + findingless: ids.filter((id) => !table[id]?.finding?.trim()), + }; +} diff --git a/tck/index.ts b/tck/index.ts index 375deb5..1f7bb75 100644 --- a/tck/index.ts +++ b/tck/index.ts @@ -18,6 +18,7 @@ */ export * from "./driver"; export * from "./scope"; +export * from "./expected"; export * from "./unverifiable"; export * from "./capabilities"; export * from "./wait"; diff --git a/tck/main.ts b/tck/main.ts index 7b14dc3..3d6b015 100644 --- a/tck/main.ts +++ b/tck/main.ts @@ -21,7 +21,16 @@ * - PARTIAL -- zero FAILs but at least one check degraded to SKIPPED * because the driver answered with assert.ts's UNVERIFIABLE sentinel. * Exit code 0: a check that could not be evaluated is not a defect. - * Only FAIL and ERROR make the process exit non-zero. + * + * The verdict is what the CSMS did; the EXIT CODE is what that means for this + * driver, and the two stopped being the same thing when expected.ts arrived. + * A FAIL the driver declared expected exits 0 -- it is a finding already + * written down, not news -- and a PASS it declared expected exits 1, because + * a list that cannot shrink is a mute. So: + * + * exit non-zero <= an undeclared FAIL/ERROR, or a declared one that passed. + * + * A driver that declares nothing keeps the original rule exactly. */ import { mkdirSync } from "node:fs"; @@ -31,10 +40,17 @@ import { AssertRecorder } from "./assert"; import { CSMS_OPERATION_ACTIONS, driverCapabilities, + driverExpectedFailures, driverScope, UnsupportedOperationError, } from "./driver"; import { loadDriverModule } from "./driver-registry"; +import { + expectedFailureCoverage, + expectedFailureFor, + type ExpectedFailureEntry, + type ExpectedFailureTable, +} from "./expected"; import { scopeCoverage, templateIdsWithStatus, @@ -129,6 +145,21 @@ async function scopeEntryFor( return (await scopeTable())?.[templateId]; } +/** + * The driver's expected-failure entry for one scenario, or undefined when it + * declared none -- in which case every failure is a failure, which is what a + * driver with nothing to declare should keep saying. + * + * Offline for the same reason {@link scopeEntryFor} is, and it matters here + * too: `ocpp-tck run` must be able to tell you a red scenario was expected + * without first demanding the credential it is not going to use. + */ +async function expectedFailureEntryFor( + templateId: string, +): Promise { + return expectedFailureFor(await expectedFailureTable(), templateId); +} + /** Result of one scenario execution -- either it produced checks, or the * driver declared the scenario undrivable partway through drive(). */ type ScenarioRun = @@ -150,6 +181,9 @@ type ScenarioRun = let driverModulePromise: Promise | undefined; let driverPartsPromise: Promise | undefined; let scopeTablePromise: Promise | undefined; +let expectedFailurePromise: + | Promise + | undefined; /** * The one environment this process shows a driver, named once so that the @@ -182,6 +216,18 @@ async function scopeTable(): Promise { return scopeTablePromise; } +/** The expected-failure list, resolved once with the SAME `ENV` as the scope + * table and as `create()`. A list describing one release line while the + * requests target the other excuses the wrong scenarios. */ +async function expectedFailureTable(): Promise< + ExpectedFailureTable | undefined +> { + expectedFailurePromise ??= driverModule().then((module) => + driverExpectedFailures(module, ENV), + ); + return expectedFailurePromise; +} + /** * Fills in the optional halves of CsmsRecords with stubs that throw * UnsupportedOperationError, carrying the driver's own reason. @@ -530,6 +576,80 @@ interface ScenarioOutcome extends RetryOutcome { * a flake. */ isolatedRetry?: RetryOutcome; + /** The driver's declaration that this scenario is known to fail, attached + * whether or not it did. Both directions are reported: a declared FAIL is + * excused, a declared PASS fails the sweep. */ + expected?: ExpectedFailureEntry; +} + +/** + * Did this scenario really fail -- after the isolated retry has had its say? + * + * ONE definition, deliberately, because there are now two consumers: the exit + * code, and the comparison against the driver's expected-failure list. Written + * twice they would drift, and the drift would be silent in the worst possible + * place -- a row excused by one rule and counted by the other. + * + * A parallel-lane FAIL/ERROR that did not fail its isolated retry is a flake, + * not a failure; see {@link retryFailedOutcomesIsolated}. + */ +function effectivelyFailed(outcome: ScenarioOutcome): boolean { + if (!isFailure(outcome.verdict)) return false; + return ( + outcome.isolatedRetry === undefined || + isFailure(outcome.isolatedRetry.verdict) + ); +} + +/** + * How an outcome lands against what the driver declared. This -- not the + * verdict -- is what the exit code reads. + * + * `expected-fail` is the whole point of the list: a finding already written + * down is not news, and muting it one scenario at a time is what lets the + * other 46 be reported at all. + * + * `unexpected-pass` is what keeps the list from becoming the job-level mute it + * replaced. It fires on a declared scenario that did NOT effectively fail, + * which includes the case where it failed in parallel and passed its isolated + * retry -- there is no "expected flaky", so an entry that passes any way at + * all is an entry to delete or to re-word. + */ +type SweepStanding = + | "ok" + | "expected-fail" + | "unexpected-fail" + | "unexpected-pass"; + +function standingOf(outcome: ScenarioOutcome): SweepStanding { + const failed = effectivelyFailed(outcome); + if (outcome.expected === undefined) { + return failed ? "unexpected-fail" : "ok"; + } + if (failed) return "expected-fail"; + // NOT APPLICABLE is unreachable here: check-driver rejects an id that is + // both declared expected-failing and NOT_APPLICABLE in the scope table, and + // the runtime UnsupportedOperationError escape would be the same + // contradiction arriving late. Reported as an unexpected pass rather than + // silently tolerated -- "it never ran" is not "it was fixed", and the + // message below says which. + return "unexpected-pass"; +} + +/** Why an unexpected pass is one, in the words a reader needs to tell a fix + * from a flake from a contradiction without re-reading this file. */ +function unexpectedPassDetail(outcome: ScenarioOutcome): string { + if (outcome.verdict === "NOT APPLICABLE") { + return "it never ran (NOT APPLICABLE), so it cannot be failing as declared"; + } + if (isFailure(outcome.verdict)) { + return ( + `it failed in the parallel lane and ${outcome.isolatedRetry?.verdict} ` + + "on its isolated retry -- either upstream fixed it, or the scenario is " + + "flaky and the flake is the bug to fix" + ); + } + return `it came back ${outcome.verdict} outright -- the finding looks fixed`; } /** @@ -544,6 +664,11 @@ async function runOneForSweep( spec: ScenarioSpec, cpId: string, ): Promise { + // Read before the run and carried on every branch, including the ones that + // never start a container: a declaration that only got attached to failures + // could never report the case the list exists to catch, which is the row + // that stopped failing. + const expected = await expectedFailureEntryFor(spec.templateId); const scope = await scopeEntryFor(spec.templateId); if (scope?.status === "NOT_APPLICABLE") { process.stderr.write( @@ -557,6 +682,7 @@ async function runOneForSweep( failed: null, skipped: null, reason: scope.reason, + expected, }; } @@ -571,6 +697,7 @@ async function runOneForSweep( failed: null, skipped: null, reason: run.reason, + expected, }; } return { @@ -580,6 +707,7 @@ async function runOneForSweep( checks: run.rec.total, failed: run.rec.failed, skipped: run.rec.skipped, + expected, }; } catch (err) { const message = @@ -595,6 +723,7 @@ async function runOneForSweep( failed: null, skipped: null, errorMessage: message, + expected, }; } } @@ -613,6 +742,12 @@ async function runOneForSweep( * --parallel's wall-clock win. * * PARTIAL and NOT APPLICABLE are never retried: neither is a failure. + * + * A DECLARED expected failure IS retried, and that is not waste. The retry is + * what turns "it failed" into "it failed with no lane contending", which is + * the only evidence that would let the declaration be deleted; a declared row + * that passes its isolated retry is reported as an unexpected pass, because + * there is no "expected flaky". */ async function retryFailedOutcomesIsolated( outcomes: ScenarioOutcome[], @@ -715,7 +850,19 @@ async function writeSummary( const checks = o.checks === null ? "-" : String(o.checks); const failed = o.failed === null ? "-" : String(o.failed); const skipped = o.skipped === null ? "-" : String(o.skipped); - const verdict = o.reason ? `${o.verdict} (${o.reason})` : o.verdict; + let verdict = o.reason ? `${o.verdict} (${o.reason})` : o.verdict; + // The declaration is rendered INTO the verdict cell rather than into a + // column of its own, so that a reader scanning for red sees, in the same + // glance, whether it was already known -- and, for an unexpected pass, + // sees the row that has to be deleted. + const standing = standingOf(o); + if (standing === "expected-fail") { + verdict = `${verdict} — EXPECTED FAIL: ${o.expected?.reason} (${o.expected?.finding})`; + } else if (standing === "unexpected-pass") { + verdict = + `${verdict} — UNEXPECTED PASS: declared expected-failing, but ` + + `${unexpectedPassDetail(o)}. Delete the entry or re-word it.`; + } const base = `| ${o.templateId} | ${o.cpId} | ${verdict} | ${checks} | ${failed} | ${skipped} |`; if (!anyRetried) return base; if (!o.isolatedRetry) return `${base} - |`; @@ -743,6 +890,36 @@ async function writeSummary( `${naCount} NOT APPLICABLE (out of scope for this CSMS). Neither fails the sweep.`, ); } + const expectedFails = outcomes.filter( + (o) => standingOf(o) === "expected-fail", + ); + const unexpectedPasses = outcomes.filter( + (o) => standingOf(o) === "unexpected-pass", + ); + if (expectedFails.length > 0) { + notes.push( + "", + `${expectedFails.length} EXPECTED FAIL — declared by this driver as a ` + + "known finding about the CSMS, so the sweep still exits 0. The scope " + + "rows stay DRIVABLE and the scenarios still ran; only the exit code " + + "changed. Each entry names where the finding is written down:", + ...expectedFails.map( + (o) => `- \`${o.templateId}\` — ${o.expected?.finding}`, + ), + ); + } + if (unexpectedPasses.length > 0) { + notes.push( + "", + `${unexpectedPasses.length} UNEXPECTED PASS — declared expected-failing ` + + "and did not fail. **This fails the sweep**, on purpose: it is how a " + + "known-red entry gets deleted when it stops being true, instead of " + + "outliving the defect it documents.", + ...unexpectedPasses.map( + (o) => `- \`${o.templateId}\` — ${unexpectedPassDetail(o)}`, + ), + ); + } if (anyRetried) { const flakeCount = outcomes.filter( (o) => o.isolatedRetry !== undefined && !isFailure(o.isolatedRetry.verdict), @@ -850,8 +1027,15 @@ async function runGroupSweep( }]` : ""; const reasonSuffix = o.reason ? ` -- ${o.reason}` : ""; + const standing = standingOf(o); + const declaredSuffix = + standing === "expected-fail" + ? ` [EXPECTED FAIL: ${o.expected?.reason} (${o.expected?.finding})]` + : standing === "unexpected-pass" + ? ` [UNEXPECTED PASS: declared expected-failing, but ${unexpectedPassDetail(o)}]` + : ""; process.stderr.write( - ` ${o.verdict}: ${o.templateId} (${o.cpId}, ${o.checks ?? "-"} checks, ${o.failed ?? "-"} failed, ${o.skipped ?? "-"} skipped)${retrySuffix}${reasonSuffix}\n`, + ` ${o.verdict}: ${o.templateId} (${o.cpId}, ${o.checks ?? "-"} checks, ${o.failed ?? "-"} failed, ${o.skipped ?? "-"} skipped)${retrySuffix}${declaredSuffix}${reasonSuffix}\n`, ); } @@ -859,11 +1043,16 @@ async function runGroupSweep( process.stderr.write(`[runner] results table: ${summaryPath}\n`); // A parallel-lane FAIL/ERROR that did not fail on its isolated retry is a - // flake, not a real failure -- it does not fail the sweep. + // flake, not a real failure -- it does not fail the sweep. Everything below + // reads that through effectivelyFailed(), via standingOf(). const badOutcomes = outcomes.filter( - (o) => - isFailure(o.verdict) && - (o.isolatedRetry === undefined || isFailure(o.isolatedRetry.verdict)), + (o) => standingOf(o) === "unexpected-fail", + ); + const expectedFails = outcomes.filter( + (o) => standingOf(o) === "expected-fail", + ); + const unexpectedPasses = outcomes.filter( + (o) => standingOf(o) === "unexpected-pass", ); const flakeCount = outcomes.filter( (o) => @@ -876,17 +1065,39 @@ async function runGroupSweep( `[runner] ${flakeCount} parallel-only flake(s) in group '${groupName}' (did not fail on isolated retry) -- see ${summaryPath}\n`, ); } + if (expectedFails.length > 0) { + process.stderr.write( + `[runner] ${expectedFails.length} scenario(s) in group '${groupName}' failed as this driver declared they would -- not a build failure; see ${summaryPath}\n`, + ); + } + // Reported BEFORE the ordinary failures and counted separately, because the + // two need opposite fixes: an unexpected failure is a defect to chase, an + // unexpected pass is a line to delete. + if (unexpectedPasses.length > 0) { + process.stderr.write( + `[runner] ${unexpectedPasses.length} scenario(s) in group '${groupName}' are declared expected-failing and did NOT fail:\n`, + ); + for (const o of unexpectedPasses) { + process.stderr.write( + ` ${o.templateId}: ${unexpectedPassDetail(o)} -- delete its expected-failure entry, or re-word it to say what is still true.\n`, + ); + } + } if (badOutcomes.length > 0) { process.stderr.write( `[runner] ${badOutcomes.length}/${outcomes.length} scenario(s) in group '${groupName}' FAILed or errored -- see ${summaryPath}\n`, ); - return 1; } + if (badOutcomes.length > 0 || unexpectedPasses.length > 0) return 1; + const partialCount = outcomes.filter((o) => o.verdict === "PARTIAL").length; const naCount = outcomes.filter((o) => o.verdict === "NOT APPLICABLE").length; process.stderr.write( - `[runner] no failures in group '${groupName}': ${outcomes.length} scenario(s), ` + + `[runner] no unexpected failures in group '${groupName}': ${outcomes.length} scenario(s), ` + `${partialCount} PARTIAL, ${naCount} NOT APPLICABLE` + + (expectedFails.length > 0 + ? `, ${expectedFails.length} EXPECTED FAIL` + : "") + (flakeCount > 0 ? `, ${flakeCount} flake(s) resolved by isolated retry` : "") + ".\n", ); @@ -1166,10 +1377,12 @@ async function checkDriver(argv: string[]): Promise { let module: CsmsDriverModule; let scope: ScopeTable | undefined; let capabilities: CsmsCapabilities | undefined; + let expected: ExpectedFailureTable | undefined; try { module = await driverModule(); scope = await scopeTable(); capabilities = driverCapabilities(module, ENV); + expected = await expectedFailureTable(); } catch (err) { process.stderr.write( `${err instanceof Error ? err.message : String(err)}\n`, @@ -1245,6 +1458,46 @@ async function checkDriver(argv: string[]): Promise { } } + // Everything about the expected-failure list that is knowable offline. What + // is NOT knowable here is whether those scenarios still fail -- only a sweep + // answers that, by reporting an UNEXPECTED PASS. + if (expected) { + const drift = expectedFailureCoverage(expected, scope, registered); + if (drift.stale.length > 0) { + problems.push( + `${drift.stale.length} expected-failure entry(ies) name a scenario ` + + `nobody registers: ${drift.stale.join(", ")}.\n` + + " -> delete the entry, or fix the templateId if the scenario was " + + "renamed. An entry under an old id excuses nothing and can never be " + + "cleaned up by a passing run.", + ); + } + if (drift.notApplicable.length > 0) { + problems.push( + `${drift.notApplicable.length} expected-failure entry(ies) are also ` + + `NOT_APPLICABLE in the scope table: ${drift.notApplicable.join(", ")}.\n` + + " -> both cannot be true. A NOT_APPLICABLE scenario never starts, " + + "so it can neither fail as declared nor ever pass and delete the entry.", + ); + } + if (drift.reasonless.length > 0) { + problems.push( + `${drift.reasonless.length} expected-failure entry(ies) carry an ` + + `empty reason: ${drift.reasonless.join(", ")}. "Known red" is the ` + + "observation, not the mechanism.", + ); + } + if (drift.findingless.length > 0) { + problems.push( + `${drift.findingless.length} expected-failure entry(ies) carry an ` + + `empty finding: ${drift.findingless.join(", ")}.\n` + + " -> name where it is written down: an upstream issue, or the row " + + "of this driver's README gap table that carries the evidence. A " + + "known-red with nowhere to read about it is how the list rots.", + ); + } + } + if (capabilities) { const known = new Set(CSMS_OPERATION_ACTIONS); const unknown = [...capabilities.operations].filter( @@ -1279,6 +1532,7 @@ async function checkDriver(argv: string[]): Promise { NOT_APPLICABLE: templateIdsWithStatus(scope, "NOT_APPLICABLE").length, } : null, + expectedFailures: expected ? Object.keys(expected).sort() : null, problems, warnings, }; @@ -1308,6 +1562,18 @@ async function checkDriver(argv: string[]): Promise { : "") + ".\n", ); + // Listed rather than counted: this is the one declaration whose whole job is + // to shrink, and a number does not tell a reviewer which rows to go and check. + if (summary.expectedFailures && summary.expectedFailures.length > 0) { + process.stderr.write( + ` ${summary.expectedFailures.length} scenario(s) declared expected-failing ` + + "-- each still runs, still prints FAIL, and fails the sweep the day it " + + "passes:\n", + ); + for (const id of summary.expectedFailures) { + process.stderr.write(` ${id}: ${expected?.[id]?.finding}\n`); + } + } return 0; } @@ -1519,13 +1785,19 @@ export async function cli(argv: string[]): Promise { return 1; } + // The same declaration the sweep reads, so that running one scenario by + // hand means what running it in the sweep means. Without this, + // `bun run e2e:smoke` -- which names an expected-failing scenario + // explicitly -- would contradict `bun run e2e` on the same commit. + const expected = await expectedFailureEntryFor(spec.templateId); + // Scope table first -- a NOT_APPLICABLE scenario never starts a container. const scope = await scopeEntryFor(spec.templateId); if (scope?.status === "NOT_APPLICABLE") { process.stderr.write( `[runner] RESULT: ${spec.templateId} NOT APPLICABLE (no container started): ${scope.reason}\n`, ); - return 0; + return unexpectedPassExit(spec.templateId, "NOT APPLICABLE", expected); } const options: RunOptions = { @@ -1538,7 +1810,42 @@ export async function cli(argv: string[]): Promise { process.stderr.write( `[runner] RESULT: ${spec.templateId} NOT APPLICABLE: ${run.reason}\n`, ); - return 0; + return unexpectedPassExit(spec.templateId, "NOT APPLICABLE", expected); } - return isFailure(verdictForRecorder(run.rec)) ? 1 : 0; + + const verdict = verdictForRecorder(run.rec); + if (isFailure(verdict)) { + if (expected) { + process.stderr.write( + `[runner] ${spec.templateId} EXPECTED FAIL: ${expected.reason} ` + + `(${expected.finding}) -- declared by this driver, exit 0.\n`, + ); + return 0; + } + return 1; + } + return unexpectedPassExit(spec.templateId, verdict, expected); +} + +/** + * Exit code for a single scenario that did NOT fail: 0 normally, 1 when the + * driver declared it expected-failing. + * + * A single run has no isolated retry, so there is no flake to distinguish + * here -- one non-failure IS the whole evidence, and the message says so + * rather than asserting the finding is fixed. + */ +function unexpectedPassExit( + templateId: string, + verdict: Verdict, + expected: ExpectedFailureEntry | undefined, +): number { + if (!expected) return 0; + process.stderr.write( + `[runner] ${templateId} UNEXPECTED PASS: this driver declares it ` + + `expected-failing (${expected.reason} -- ${expected.finding}), and it ` + + `came back ${verdict}. Delete the expected-failure entry, or re-word it ` + + "to say what is still true.\n", + ); + return 1; } diff --git a/tests/driver-env-scope.ts b/tests/driver-env-scope.ts index a093448..7245f8d 100644 --- a/tests/driver-env-scope.ts +++ b/tests/driver-env-scope.ts @@ -1,9 +1,9 @@ // Copyright 2026 Julien Herr // SPDX-License-Identifier: Apache-2.0 /** - * driver-env-scope.ts -- `scope` and `capabilities` describe the server the - * driver was RESOLVED for, not the one that happened to be in `process.env` - * when the module was imported. + * driver-env-scope.ts -- `scope`, `capabilities` and `expectedFailures` + * describe the server the driver was RESOLVED for, not the one that happened + * to be in `process.env` when the module was imported. * * WHY THIS IS TYPESCRIPT AND NOT A SHELL GUARD. Every other check in this * directory drives the CLI, and the CLI always passes `process.env` -- which @@ -34,10 +34,12 @@ import { csmsDriver as steve } from "../drivers/steve/index"; import { STEVE_SCOPE } from "../drivers/steve/scope"; import { driverCapabilities, + driverExpectedFailures, driverScope, type CsmsDriverModule, type CsmsEnv, } from "../tck/driver"; +import type { ExpectedFailureTable } from "../tck/expected"; const failures: string[] = []; @@ -63,6 +65,21 @@ const SYNTHETIC: CsmsDriverModule = { reservations: false, chargingProfiles: false, }), + // The third declaration, and the one where getting the env wrong is worst: + // a list resolved for the other release line excuses scenarios that are + // failing for reasons nobody has looked at, AND withholds the UNEXPECTED + // PASS that would delete the entries that no longer apply. Both directions + // are silent, which is why it is asserted here rather than left to + // check-driver. + expectedFailures: (env): ExpectedFailureTable => + env.LINE === "old" + ? { + "one-scenario": { + reason: "the old line has the defect", + finding: "synthetic", + }, + } + : {}, create() { throw new Error("resolving a declaration must not call create()"); }, @@ -71,7 +88,14 @@ const SYNTHETIC: CsmsDriverModule = { const PLAIN_SCOPE = { "one-scenario": { status: "DRIVABLE", reason: "stated once" }, } as const; -const PLAIN: CsmsDriverModule = { ...SYNTHETIC, scope: PLAIN_SCOPE }; +const PLAIN_EXPECTED = { + "one-scenario": { reason: "stated once", finding: "synthetic" }, +} as const; +const PLAIN: CsmsDriverModule = { + ...SYNTHETIC, + scope: PLAIN_SCOPE, + expectedFailures: PLAIN_EXPECTED, +}; check( driverScope(SYNTHETIC, { LINE: "old" })?.["one-scenario"]?.status === @@ -84,11 +108,32 @@ check( driverCapabilities(SYNTHETIC, {})?.operations.has("Reset") === true, "driverCapabilities() does not answer from the env it was handed.", ); +check( + driverExpectedFailures(SYNTHETIC, { LINE: "old" })?.["one-scenario"] !== + undefined && driverExpectedFailures(SYNTHETIC, {})?.["one-scenario"] === undefined, + "driverExpectedFailures() does not answer from the env it was handed.", +); check( driverScope(PLAIN, { LINE: "old" }) === PLAIN_SCOPE, "a plain ScopeTable no longer resolves to itself -- the union broke the " + "single-release driver, which is the common case.", ); +check( + driverExpectedFailures(PLAIN, { LINE: "old" }) === PLAIN_EXPECTED, + "a plain ExpectedFailureTable no longer resolves to itself.", +); +// The field is OPTIONAL, and a driver that declares nothing must keep the +// original rule -- every failure is a failure. Resolving `undefined` to +// anything else would silently excuse scenarios for every driver that never +// opted in. +check( + driverExpectedFailures( + { ...SYNTHETIC, expectedFailures: undefined }, + { LINE: "old" }, + ) === undefined, + "a driver that declares no expected failures no longer resolves to " + + "undefined, so 'every failure is a failure' stopped being the default.", +); // --- the bundled drivers use it --------------------------------------------- const V1_ENV: CsmsEnv = { CITRINE_VARIANT: "v1" }; @@ -160,16 +205,17 @@ check( if (failures.length > 0) { process.stderr.write( - "FAIL: a driver's scope or capabilities do not follow the environment " + - "they are resolved with.\n", + "FAIL: a driver's scope, capabilities or expected failures do not follow " + + "the environment they are resolved with.\n", ); for (const failure of failures) process.stderr.write(` - ${failure}\n`); process.exit(1); } process.stdout.write( - "Driver scope and capabilities follow the resolved environment " + - `(${V1_LOCAL_LIST_SCENARIOS.length} rows and ` + + "Driver scope, capabilities and expected failures follow the resolved " + + `environment (${V1_LOCAL_LIST_SCENARIOS.length} rows and ` + `${LOCAL_LIST_ACTIONS.length} operations differ between the two ` + - "CitrineOS lines, and a plain table still resolves to itself).\n", + "CitrineOS lines; a plain table still resolves to itself, and an absent " + + "expected-failure list still resolves to undefined).\n", ); diff --git a/types/tck/driver.d.ts b/types/tck/driver.d.ts index 352a719..42ee3b1 100644 --- a/types/tck/driver.d.ts +++ b/types/tck/driver.d.ts @@ -51,6 +51,7 @@ * 3. Branch on a scenario id. A driver that wants per-scenario behaviour is * describing a capability gap; declare it in the driver's scope table. */ +import type { ExpectedFailureTable } from "./expected"; import type { ScopeTable } from "./scope"; /** A CSMS-side transaction / charging-session handle. `""` = none. */ export type TransactionRef = string; @@ -416,6 +417,23 @@ export interface CsmsDriverModule { * credentials, printed by `ocpp-tck check-driver`, and equally free to be a * function of the environment. Resolve it with {@link driverCapabilities}. */ readonly capabilities?: EnvDependent; + /** + * Scenarios this CSMS is KNOWN to fail -- drivable, run, red, and understood. + * + * The complement of {@link CsmsDriverModule.scope}, not a part of it: a + * listed scenario keeps its DRIVABLE row, still starts a container and still + * prints FAIL. What the list changes is only the sweep's exit code, so that + * a job muted for one finding can still report every other scenario. An + * entry that PASSES fails the sweep in the other direction -- see + * {@link ./expected}. + * + * Absent = "every failure is a failure", which is what a driver with nothing + * to declare should keep saying. Free to be a function of the environment + * for the same reason `scope` is: a CSMS with two release lines does not + * have the same defects on both. Resolve it with + * {@link driverExpectedFailures}. + */ + readonly expectedFailures?: EnvDependent; /** Called once per process, and the result is shared by every parallel lane, * so what it returns must be safe to use concurrently and must hold no * per-lane state. The contract used to promise one instance per lane, which @@ -444,3 +462,9 @@ export interface CsmsDriverModule { export declare function driverScope(module: CsmsDriverModule, env: CsmsEnv): ScopeTable | undefined; /** {@link driverScope} for the capability declaration. */ export declare function driverCapabilities(module: CsmsDriverModule, env: CsmsEnv): CsmsCapabilities | undefined; +/** + * {@link driverScope} for the expected-failure list, and it must be resolved + * with the SAME env: a list naming defects of one release line, applied to a + * sweep pointed at the other, excuses the wrong scenarios in both directions. + */ +export declare function driverExpectedFailures(module: CsmsDriverModule, env: CsmsEnv): ExpectedFailureTable | undefined; diff --git a/types/tck/expected.d.ts b/types/tck/expected.d.ts new file mode 100644 index 0000000..191d511 --- /dev/null +++ b/types/tck/expected.d.ts @@ -0,0 +1,79 @@ +/** + * expected.ts -- the TYPE of a driver's expected-failure list, and the drift + * checks that keep it honest. The DATA lives in each driver + * (drivers//expected.ts): which scenarios a CSMS gets wrong is a fact about + * that CSMS, not about this harness. + * + * WHY THIS IS NOT A SCOPE STATUS. {@link ./scope} answers "can this CSMS drive + * the scenario at all?", and its rules forbid demoting a row to + * NOT_APPLICABLE to make a red scenario go away -- that converts a finding + * about the CSMS into a silence about the harness. This file answers the other + * question: the scenario IS drivable, it runs, it prints FAIL, and we already + * know why. The row stays DRIVABLE; only the exit code changes. + * + * WHAT IT REPLACES. Before it, the only vocabulary for "we know this one is + * red" was `continue-on-error` on the whole CI job -- job-granular where the + * intent is scenario-granular. A broken compose file, a failed provisioning + * step and a regression from one failure to twenty were all as green as the + * finding the job was muted for, so the job could not report the one thing it + * existed to report: every OTHER scenario. + * + * THE LIST HAS TO BE ABLE TO SHRINK, or it is just a quieter mute. So an + * expected failure that PASSES is itself a failure -- reported as + * UNEXPECTED PASS, with a non-zero exit -- which is how a row gets deleted + * when upstream fixes the defect, instead of outliving it. + * + * RULES FOR A DRIVER AUTHOR EDITING ITS LIST + * - `reason` names the mechanism, cited, the way a scope row's does. "Fails + * on this CSMS" is not a reason; it is the observation being explained. + * - `finding` says where the finding is written down -- an upstream issue, or + * the row of the driver README's gap table that carries the evidence. A + * known-red with nowhere to read about it is a claim nobody can review, and + * it is the shape this list rots into if the field is optional. + * - NEVER add a row to quiet a flake. A scenario that sometimes passes has a + * timing bug, and the fix is the scenario. There is deliberately no + * "expected flaky" status here: it would re-create the job-level mute one + * scenario at a time. + */ +import type { ScopeTable } from "./scope"; +export interface ExpectedFailureEntry { + /** The mechanism, cited. Not "known red" -- WHY it is red. */ + reason: string; + /** + * Where the finding is recorded: an upstream issue URL, or the driver + * README's gap-table row. Required, and checked non-empty by + * `ocpp-tck check-driver`. + */ + finding: string; +} +/** One entry per scenario `templateId` this CSMS is known to fail. */ +export type ExpectedFailureTable = Readonly>; +export declare function expectedFailureFor(table: ExpectedFailureTable | undefined, templateId: string): ExpectedFailureEntry | undefined; +/** + * Everything about a list that can be established WITHOUT running anything. + * + * Four kinds, and each is a way the list stops meaning what it says: + * + * `stale` -- an id nobody registers. Usually a rename, and it silently stops + * excusing anything; the scenario it was written for now fails the build under + * its new name, which at least is loud. The reverse of the same rename is not: + * a row kept under the OLD id while the scenario is fixed under the new one + * never gets the UNEXPECTED PASS that would delete it. + * + * `notApplicable` -- the scope table says the scenario never runs on this + * CSMS, and this list says it runs and fails. Both cannot be true. Left + * unchecked it is the quiet way a driver ends up with a row that can never + * fire and can never be cleaned up, because the only thing that deletes a row + * is the scenario passing, and a scenario that never starts never passes. + * + * `reasonless` / `findingless` -- see the rules in this file's header. + * + * What it deliberately CANNOT check is whether the scenario actually fails. + * Only a sweep knows that, and it is the sweep that reports UNEXPECTED PASS. + */ +export declare function expectedFailureCoverage(table: ExpectedFailureTable, scope: ScopeTable | undefined, registeredTemplateIds: readonly string[]): { + stale: string[]; + notApplicable: string[]; + reasonless: string[]; + findingless: string[]; +}; diff --git a/types/tck/index.d.ts b/types/tck/index.d.ts index e8e47bd..496d387 100644 --- a/types/tck/index.d.ts +++ b/types/tck/index.d.ts @@ -16,6 +16,7 @@ */ export * from "./driver"; export * from "./scope"; +export * from "./expected"; export * from "./unverifiable"; export * from "./capabilities"; export * from "./wait"; From ae17a6a6fb47357e098e894e8d3806f570c9b10a Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 14 Aug 2026 09:39:25 +0200 Subject: [PATCH 02/11] feat(citrineos): declare the Blocked finding as an expected failure cert16-tc023-3-authorize-blocked fails deterministically against the pinned image -- CitrineOS answers Invalid where OCPP 1.6 requires Blocked, 3 runs out of 3 -- and tck/scope.ts forbids making it green by demoting the row. Until now the only alternative was muting the whole CI job. The mechanism sentence moves to expected.ts and scope.ts imports it, the way variant.ts already owns NO_RESERVATIONS for both scope.ts and requests.ts. The two halves a reader compares -- "this row is drivable" and "this row is expected to fail, and here is why" -- must not be free to disagree. The v1.9.1 list is deliberately EMPTY, and the emptiness is a claim about evidence rather than about CitrineOS. v1 fails 16 of 47, nearly all through upstream citrineos/citrineos#160, but that measurement is a 2026-08-11 snapshot and no CI job runs the line -- so nothing would ever report one of those entries as an UNEXPECTED PASS and delete it. A list no run can shrink is the rot this mechanism exists to replace. On v1 every failure stays a failure. --- VENDOR.md | 1 + drivers/citrineos/README.md | 20 ++++++- drivers/citrineos/expected.ts | 75 +++++++++++++++++++++++++++ drivers/citrineos/index.ts | 5 ++ drivers/citrineos/scope.ts | 32 +++++++----- types/drivers/citrineos/expected.d.ts | 32 ++++++++++++ types/drivers/citrineos/scope.d.ts | 12 +++-- 7 files changed, 156 insertions(+), 21 deletions(-) create mode 100644 drivers/citrineos/expected.ts create mode 100644 types/drivers/citrineos/expected.d.ts diff --git a/VENDOR.md b/VENDOR.md index 9169b28..a5719fd 100644 --- a/VENDOR.md +++ b/VENDOR.md @@ -85,6 +85,7 @@ strongest available demonstration that the core names no CSMS. | `drivers/citrineos/profiles.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `drivers/citrineos/records.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `drivers/citrineos/scope.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | +| `drivers/citrineos/expected.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `drivers/citrineos/variant.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `drivers/citrineos/provision.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | diff --git a/drivers/citrineos/README.md b/drivers/citrineos/README.md index e50fcc3..0b87513 100644 --- a/drivers/citrineos/README.md +++ b/drivers/citrineos/README.md @@ -26,6 +26,14 @@ in the scope table: `tck/scope.ts` forbids demoting a row to `NOT_APPLICABLE` to make a red scenario go away, and a TCK whose second driver reports 100% green is a TCK that has stopped measuring. It is below. +It is **declared**, in [`expected.ts`](expected.ts), which is why the CI job +that runs this driver is blocking rather than `continue-on-error`. The +scenario still runs and still prints `FAIL`; the sweep reports it as +`EXPECTED FAIL` and exits 0, so a *new* red still fails the build. The day +upstream fixes the `Blocked` mapping, that row comes back `UNEXPECTED PASS` +and fails the build until the entry is deleted — which is how the list shrinks +instead of rotting. + The count was 38 / 7 / 2 on 2026-08-11. The second failure was TC_044.2, and it was **ours**: the scenario asked for a retrieveDate +90s against a 110s hold, leaving ~20s for the status train. That is fixed in `tck/specs/firmware.ts`, so @@ -146,6 +154,14 @@ and the database holds `Connectors: 0` after the whole sweep: that is isolated retry; there were no flakes. Nothing was reported upstream, because it is already fixed where it matters. +**None of those sixteen is declared in [`expected.ts`](expected.ts)**, and the +omission is about evidence rather than about CitrineOS: no CI job runs this +line, so nothing would ever report one of those entries as an `UNEXPECTED PASS` +and delete it. A list no run can shrink is the rot the mechanism exists to +replace — it would read as sixteen reviewed findings while being one stale +snapshot. On v1 every failure stays a failure; whoever puts the line back under +a sweep gets the honest list from the run. + `tc023-3` fails identically on both lines, which is the useful control: it is a property of the `Authorize` handler, not of either release. @@ -259,10 +275,10 @@ a `reason` that cannot name the limitation is `CONDITIONAL`, not | **No OCPP 1.6 reservation endpoints.** No `@AsMessageEndpoint` binds `ReserveNow` or `CancelReservation` to `OCPPVersion.OCPP1_6`. The 1.6 schemas exist, the `Reservations` table exists, `evdriver.responses` lists both actions — nothing routes them, and no 1.6 response handler exists either. | 7 scenarios `NOT APPLICABLE`; the driver omits `records.reservations` entirely. | Verified at `v1.9.1`, `v2.0.0-beta1` and `main`. | | **Local auth list is v2-only.** `EVDriverOcpp16Api` gained `sendLocalList` / `getLocalListVersion` in the v2 line. | 6 scenarios, drivable only on the pinned prerelease. | `packages/core/src/modules/EVDriver/src/module/1.6/MessageApi.ts` | | **No charging-profile registry.** `ChargingProfiles` has no `description` or `name` column, and nothing to look one up by. | `refByDescription` resolves from this driver's own catalogue instead. Not a scenario cost: OCPP 1.6 carries the profile inline. | `packages/core/src/dal/layers/drizzle/schema/ChargingProfile.ts`, and [`profiles.ts`](profiles.ts) | -| **`Blocked` is unreachable from the 1.6 `Authorize` path.** The handler reaches its status mapper only through the `status === Accepted` branch, so a stored `Blocked` falls through to the default `Invalid`. The only route to a real `Blocked` is an `IAuthorizer`, and the container registers `authorizers: asValue([])` with no setting that changes it. | TC_023.3 — see `scope.ts` for its current status. | `AuthorizeRequestOcpp16Handler.ts`, `apps/ocpp-server/src/container.ts` | +| **`Blocked` is unreachable from the 1.6 `Authorize` path.** The handler reaches its status mapper only through the `status === Accepted` branch, so a stored `Blocked` falls through to the default `Invalid`. The only route to a real `Blocked` is an `IAuthorizer`, and the container registers `authorizers: asValue([])` with no setting that changes it. | TC_023.3 — `scope.ts` keeps the row DRIVABLE, `expected.ts` declares the red. | `AuthorizeRequestOcpp16Handler.ts`, `apps/ocpp-server/src/container.ts` | | **No REST for `Authorizations`.** `EVDriverDataApi` exposes exactly one route, a read-only local-list-version GET. | `driver provision` writes fixtures through GraphQL. | [`provision.ts`](provision.ts) | | **Four foreign keys reference `Authorizations`, none cascading**: `Transactions.authorizationId`, `LocalListAuthorizations.authorizationId`, `LocalListAuthorizations.groupAuthorizationId`, and the self-reference `Authorizations.groupAuthorizationId`. | `teardown` derives its guards from the foreign keys Hasura reports instead of listing them, so a fifth on a future CitrineOS is picked up rather than aborting the whole delete. Guarding only the first was measured to leave *every* fixture in place, because psql ran the script in one implicit transaction. | Read from the foreign keys Hasura derives; see `references()` in [`provision.ts`](provision.ts). | -| **`Blocked` is unreachable from the 1.6 `Authorize` path** *(see the row above for the mechanism)*. | **TC_023.3 fails**, deterministically: CitrineOS answers `{"idTagInfo":{"status":"Invalid"}}` where the scenario requires `Blocked`. Observed 3 runs out of 3. | Kept red on purpose — see `scope.ts`. | +| **`Blocked` is unreachable from the 1.6 `Authorize` path** *(see the row above for the mechanism)*. | **TC_023.3 fails**, deterministically: CitrineOS answers `{"idTagInfo":{"status":"Invalid"}}` where the scenario requires `Blocked`. Observed 3 runs out of 3. | Kept red on purpose, and declared: `expected.ts` makes it an `EXPECTED FAIL` so the job stays blocking, and an `UNEXPECTED PASS` the day it is fixed. | | **No 1.6 request handler for `FirmwareStatusNotification`.** Every one the charge point sends is answered with `[4,…,"NotSupported","No handler found for action: FirmwareStatusNotification at module configuration"]` — 9 across the three TC_044 logs, and the only CALLERROR the CSMS emits anywhere in the suite. | **A non-conformance, and one this suite does not detect.** OCA `TC_044_{1,2,3}_CSMS` put steps 4 and 6 on the Central System — *"The Central responds with a FirmwareStatusNotification.conf"* — and a CALLERROR is not that conf. Our scenarios pass because they assert only on the statuses the charge point *sent*. See below. | `packages/core/src/handlers/requests/1.6/` — `DiagnosticsStatusNotification` has one, `FirmwareStatusNotification` does not. No ticket upstream. | | **No 1.6 response handler for `UnlockConnector` or `UpdateFirmware`.** The Calls are routed and sent; the CallResults are answered with the same `NotSupported` CALLERROR. | Harmless — the six affected scenarios all pass. | `packages/core/src/handlers/responses/1.6/` | | **`GetConfiguration` is batched server-side.** The endpoint splits a request into batches of the station's stored `GetConfigurationMaxKeys`. | *Not* a problem in practice: an unprovisioned station has no such value, so the request stays one `GetConfiguration` on the wire and both TC_019 scenarios pass. Listed because provisioning that key would change it. | `Configuration/src/module/1.6/MessageApi.ts` | diff --git a/drivers/citrineos/expected.ts b/drivers/citrineos/expected.ts new file mode 100644 index 0000000..c6db2b8 --- /dev/null +++ b/drivers/citrineos/expected.ts @@ -0,0 +1,75 @@ +// Copyright 2026 Julien Herr +// SPDX-License-Identifier: Apache-2.0 +/** + * expected.ts -- the scenarios CitrineOS is known to fail, and where each + * finding is written down. + * + * The counterpart of scope.ts, not a part of it. A row here is DRIVABLE, runs, + * starts a container and prints FAIL; what it does not do is end the build. + * tck/scope.ts forbids demoting such a row to NOT_APPLICABLE -- that would turn + * a finding about CitrineOS into a silence about the harness -- and before this + * file the only alternative was `continue-on-error` on the whole CI job, which + * muted the other 46 scenarios along with the one finding. + * + * THE MECHANISM SENTENCE LIVES HERE and scope.ts imports it, the same way + * variant.ts owns NO_RESERVATIONS for both scope.ts and requests.ts. The two + * halves a reader compares -- "this row is drivable, and here is what the CSMS + * does with it" and "this row is expected to fail, and here is why" -- must not + * be free to disagree. + */ +import type { ExpectedFailureTable } from "../../tck/expected"; +import type { CitrineVariant } from "./variant"; + +/** + * Why a stored `Blocked` idTag comes back `Invalid`, worded once. + * + * Read in the sources and confirmed on the running image, 3 runs out of 3: + * `AuthorizeRequestOcpp16Handler` reaches its status mapper only through the + * `status === Accepted` branch, so a stored `Blocked` falls through to the + * default `Invalid`. The only route to a real `Blocked` is an `IAuthorizer`, + * and `container.ts` registers `authorizers: asValue([])` with no setting that + * changes it. + */ +export const BLOCKED_UNREACHABLE = + "CitrineOS reaches its Authorize status mapper only through the " + + "`status === Accepted` branch of AuthorizeRequestOcpp16Handler, so a stored " + + "Blocked falls through to the default Invalid. The only route to a real " + + "Blocked is an IAuthorizer, and container.ts registers " + + "`authorizers: asValue([])` with no setting that changes it."; + +const V2_EXPECTED_FAILURES: ExpectedFailureTable = { + "cert16-tc023-3-authorize-blocked": { + reason: + `${BLOCKED_UNREACHABLE} The scenario requires Blocked and gets ` + + '{"idTagInfo":{"status":"Invalid"}} -- deterministic, 3 runs out of 3. ' + + "A finding against CitrineOS, not a gap in this driver.", + finding: + "drivers/citrineos/README.md, gap table row \"Blocked is unreachable " + + 'from the 1.6 Authorize path". OCA TC_023.3 marks the behaviour M for ' + + "the Central System. No upstream ticket.", + }, +}; + +/** + * DELIBERATELY EMPTY for the v1.9.1 line, and the emptiness is a claim about + * evidence rather than about CitrineOS. + * + * v1 fails far more than v2 -- 16 of 47 on 2026-08-11, nearly all of them + * through upstream citrineos/citrineos#160, which scope.ts's V1_KNOWN + * describes. But that measurement has not been repeated since, and CI never + * runs this line, so nothing would ever report one of those entries as an + * UNEXPECTED PASS. An expected-failure list that no run can shrink is exactly + * the rot this mechanism exists to replace: it would read as sixteen reviewed + * findings while being one stale snapshot. + * + * So on v1 every failure stays a failure. Whoever puts that line back under a + * sweep gets the honest list from the run, and can fill this in from it. + */ +const V1_EXPECTED_FAILURES: ExpectedFailureTable = {}; + +/** The expected-failure list for a declared variant. See variant.ts. */ +export function citrineosExpectedFailures( + variant: CitrineVariant, +): ExpectedFailureTable { + return variant === "v2" ? V2_EXPECTED_FAILURES : V1_EXPECTED_FAILURES; +} diff --git a/drivers/citrineos/index.ts b/drivers/citrineos/index.ts index 12566f9..dca1744 100644 --- a/drivers/citrineos/index.ts +++ b/drivers/citrineos/index.ts @@ -55,6 +55,7 @@ import { teardownCommand, verifyCommand, } from "./provision"; +import { citrineosExpectedFailures } from "./expected"; import { CitrineRecords } from "./records"; import { toCitrineRequest } from "./requests"; import { citrineosScope } from "./scope"; @@ -124,6 +125,10 @@ export const csmsDriver: CsmsDriverModule = { displayName: "CitrineOS", scope: (env) => citrineosScope(resolveVariant(env)), capabilities: (env) => capabilitiesFor(resolveVariant(env)), + // A function of the environment for the same reason the two above are: which + // line this driver is pointed at decides which defects it meets. See + // expected.ts for why the v1 list is empty rather than sixteen rows long. + expectedFailures: (env) => citrineosExpectedFailures(resolveVariant(env)), create(env: CsmsEnv): CsmsDriverParts { const cfg = defaultCitrineConfig(env); const records = new CitrineRecords(cfg); diff --git a/drivers/citrineos/scope.ts b/drivers/citrineos/scope.ts index ab2febb..f06446d 100644 --- a/drivers/citrineos/scope.ts +++ b/drivers/citrineos/scope.ts @@ -15,14 +15,17 @@ * 18 `/ocpp/1.6/` paths with neither `reserveNow` nor `cancelReservation` * among them. * - * TWO ROWS ARE DRIVABLE AND CURRENTLY RED, ON PURPOSE. `tck/scope.ts` forbids + * ONE ROW IS DRIVABLE AND CURRENTLY RED, ON PURPOSE. `tck/scope.ts` forbids * demoting a row to NOT_APPLICABLE to make a red scenario go away, because - * that converts a finding about the CSMS into a silence about the harness. - * Both are findings against CitrineOS rather than gaps in this driver, and - * both are named in drivers/citrineos/README.md's gap table. A TCK whose - * second driver reports 100% green is a TCK that has stopped measuring. + * that converts a finding about the CSMS into a silence about the harness. It + * is a finding against CitrineOS rather than a gap in this driver, it is named + * in drivers/citrineos/README.md's gap table, and the red itself is declared in + * expected.ts -- which is what lets a CI job stay blocking while carrying a + * known finding. A TCK whose second driver reports 100% green is a TCK that has + * stopped measuring. */ import type { ScopeEntry, ScopeTable } from "../../tck/scope"; +import { BLOCKED_UNREACHABLE } from "./expected"; import { NO_LOCAL_LIST, NO_RESERVATIONS, @@ -92,16 +95,17 @@ const V2_SCOPE = { "expiry only inside its Accepted branch -- a row stored as Expired would " + "answer Invalid.", ), + // DRIVABLE AND CURRENTLY RED. The row stays DRIVABLE because demoting it + // would hide a finding (see tck/scope.ts); the red itself is declared in + // expected.ts, which is what lets CI report the other 46 scenarios without + // muting the job. The mechanism is stated there and imported here so the two + // cannot drift. "cert16-tc023-3-authorize-blocked": d( - "DRIVABLE AND CURRENTLY RED, deliberately: CitrineOS answers " + - '{"idTagInfo":{"status":"Invalid"}} where the scenario requires Blocked ' + - "(observed 3 runs out of 3). AuthorizeRequestOcpp16Handler reaches its " + - "status mapper only through the `status === Accepted` branch, so a stored " + - "Blocked falls through to the default Invalid; the only route to a real " + - "Blocked is an IAuthorizer, and container.ts registers " + - "`authorizers: asValue([])` with no setting that changes it. This is a " + - "finding against CitrineOS, not a gap in this driver -- demoting it to " + - "NOT_APPLICABLE would hide it. See tck/scope.ts.", + `${BLOCKED_UNREACHABLE} Expressible and driven: the driver sends the ` + + "Authorize the scenario asks for and reads the answer back. What comes " + + "back is Invalid where OCPP 1.6 requires Blocked, which is a finding " + + "against CitrineOS rather than a gap in this driver -- declared in " + + "expected.ts.", ), // --- RemoteTrigger ------------------------------------------------------- diff --git a/types/drivers/citrineos/expected.d.ts b/types/drivers/citrineos/expected.d.ts new file mode 100644 index 0000000..46ae777 --- /dev/null +++ b/types/drivers/citrineos/expected.d.ts @@ -0,0 +1,32 @@ +/** + * expected.ts -- the scenarios CitrineOS is known to fail, and where each + * finding is written down. + * + * The counterpart of scope.ts, not a part of it. A row here is DRIVABLE, runs, + * starts a container and prints FAIL; what it does not do is end the build. + * tck/scope.ts forbids demoting such a row to NOT_APPLICABLE -- that would turn + * a finding about CitrineOS into a silence about the harness -- and before this + * file the only alternative was `continue-on-error` on the whole CI job, which + * muted the other 46 scenarios along with the one finding. + * + * THE MECHANISM SENTENCE LIVES HERE and scope.ts imports it, the same way + * variant.ts owns NO_RESERVATIONS for both scope.ts and requests.ts. The two + * halves a reader compares -- "this row is drivable, and here is what the CSMS + * does with it" and "this row is expected to fail, and here is why" -- must not + * be free to disagree. + */ +import type { ExpectedFailureTable } from "../../tck/expected"; +import type { CitrineVariant } from "./variant"; +/** + * Why a stored `Blocked` idTag comes back `Invalid`, worded once. + * + * Read in the sources and confirmed on the running image, 3 runs out of 3: + * `AuthorizeRequestOcpp16Handler` reaches its status mapper only through the + * `status === Accepted` branch, so a stored `Blocked` falls through to the + * default `Invalid`. The only route to a real `Blocked` is an `IAuthorizer`, + * and `container.ts` registers `authorizers: asValue([])` with no setting that + * changes it. + */ +export declare const BLOCKED_UNREACHABLE: string; +/** The expected-failure list for a declared variant. See variant.ts. */ +export declare function citrineosExpectedFailures(variant: CitrineVariant): ExpectedFailureTable; diff --git a/types/drivers/citrineos/scope.d.ts b/types/drivers/citrineos/scope.d.ts index a8c7bb2..d32e868 100644 --- a/types/drivers/citrineos/scope.d.ts +++ b/types/drivers/citrineos/scope.d.ts @@ -13,12 +13,14 @@ * 18 `/ocpp/1.6/` paths with neither `reserveNow` nor `cancelReservation` * among them. * - * TWO ROWS ARE DRIVABLE AND CURRENTLY RED, ON PURPOSE. `tck/scope.ts` forbids + * ONE ROW IS DRIVABLE AND CURRENTLY RED, ON PURPOSE. `tck/scope.ts` forbids * demoting a row to NOT_APPLICABLE to make a red scenario go away, because - * that converts a finding about the CSMS into a silence about the harness. - * Both are findings against CitrineOS rather than gaps in this driver, and - * both are named in drivers/citrineos/README.md's gap table. A TCK whose - * second driver reports 100% green is a TCK that has stopped measuring. + * that converts a finding about the CSMS into a silence about the harness. It + * is a finding against CitrineOS rather than a gap in this driver, it is named + * in drivers/citrineos/README.md's gap table, and the red itself is declared in + * expected.ts -- which is what lets a CI job stay blocking while carrying a + * known finding. A TCK whose second driver reports 100% green is a TCK that has + * stopped measuring. */ import type { ScopeTable } from "../../tck/scope"; import { type CitrineVariant } from "./variant"; From 5c133d360ab1d16df6614fb964458bab5d0c4cc9 Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 14 Aug 2026 09:39:30 +0200 Subject: [PATCH 03/11] ci: make the CitrineOS conformance sweep blocking The one red scenario is now declared in drivers/citrineos/expected.ts, so the sweep exits 0 for it and non-zero for anything else. continue-on-error was protecting one finding at the cost of every other signal the job produces: a broken compose.yaml, a failed provision, a verify mismatch and a jump from one failure to twenty were all equally green. The comment that said "turn it blocking once both rows are green" was an aspiration with no mechanism; it now describes one. Closes #10 --- .github/workflows/ci.yml | 33 ++++++++++++++++----------------- 1 file changed, 16 insertions(+), 17 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d36e063..5415d50 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -179,27 +179,26 @@ jobs: # CSMS-neutrality claim that only one CSMS ever exercises is a claim, not a # check. # - # continue-on-error, and NOT because the job is unreliable: it is expected to - # be red. Two of the 47 scenarios fail against the pinned CitrineOS, and both - # are kept red deliberately rather than demoted in the scope table -- - # tck/scope.ts forbids demoting a row to make a red scenario go away, and - # these are findings about CitrineOS: + # BLOCKING, while still carrying a known finding. One of the 47 scenarios + # fails against the pinned CitrineOS -- cert16-tc023-3-authorize-blocked, + # deterministic (3 runs of 3): CitrineOS answers Invalid where OCPP 1.6 + # requires Blocked. It is kept red rather than demoted in the scope table, + # which tck/scope.ts forbids, and it is DECLARED in + # drivers/citrineos/expected.ts, so the sweep reports it as EXPECTED FAIL and + # exits 0. # - # cert16-tc023-3-authorize-blocked deterministic (3 runs of 3): CitrineOS - # answers Invalid where OCPP 1.6 requires - # Blocked. - # cert16-tc044-2-firmware-download-failed - # flaky (1 pass in 3), thinnest timing - # margin of any scenario. - # - # So this job's value is the OTHER 45, and the artifact it uploads. Turn it - # blocking only once both rows are green -- i.e. once upstream fixes the - # Blocked mapping -- and not merely because it has been quiet for a while. - # drivers/citrineos/README.md carries the full gap table. + # This job used to be continue-on-error for that one row, and that is what + # the declaration replaced. continue-on-error is job-granular where the + # intent is scenario-granular: a broken compose.yaml, a failed provision, a + # verify mismatch and a regression from 1 failure to 20 were all as green as + # the finding it was muted for, so the job could not report the one thing it + # exists to report -- the other 46. Now a new red fails the build, and the + # day upstream fixes the Blocked mapping that row fails as UNEXPECTED PASS + # until its entry is deleted. drivers/citrineos/README.md carries the full + # gap table. e2e-citrineos: runs-on: ubuntu-latest needs: check - continue-on-error: true timeout-minutes: 45 env: CSMS_DRIVER: ./drivers/citrineos/index.ts From 0759d86d6afadb7db39bef64151d2e32127bc2ce Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 14 Aug 2026 09:41:03 +0200 Subject: [PATCH 04/11] ci: drive both CSMS from one matrix job e2e-steve and e2e-citrineos differed by a driver id, a compose path, an artifact name and one URL. They are now one `e2e` job over `matrix.driver: [steve, citrineos]`. Only possible now that both are blocking: a matrix carries one continue-on-error for the whole strategy, so the previous arrangement -- one lane muted for a known finding, one not -- could not have been expressed here. fail-fast is off, explicitly. The default cancels the sibling on the first failure, and these two lanes exist precisely because each answers a question the other cannot; a cancelled lane is a missing answer, not a saved minute. Both CSMS URLs are set for both lanes rather than threaded through matrix.include. Each driver reads only its own, so the unused one is inert, and listing them beats hiding which settings exist. The reasoning the two jobs carried -- why SteVe, why CitrineOS, why the whole sweep on every event, why --retry-failed-isolated, why `bun run e2e` rather than `run-all` -- is merged above the matrix rather than dropped. A matrix that costs the record of why two CSMS are driven at all would be a bad trade. NOTE FOR THE REPOSITORY SETTINGS: the required-check names change from `e2e-steve` / `e2e-citrineos` to `e2e (steve)` / `e2e (citrineos)`. --- .github/workflows/ci.yml | 182 +++++++++++++++------------------------ 1 file changed, 69 insertions(+), 113 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5415d50..02e46ba 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -79,42 +79,78 @@ jobs: - name: Shell guards lint clean run: shellcheck tests/*.sh tools/*.sh - # Everything above is offline and proves the drivers type-check. The two e2e - # jobs prove they work: real containers, real OCPP-J frames. They are peers, - # and each answers a question the other cannot -- see e2e-citrineos below. + # Everything above is offline and proves the drivers type-check. This job + # proves they WORK: real containers, real OCPP-J frames. It runs once per + # bundled driver, and the two lanes are peers -- each answers a question the + # other cannot. # - # This one: SteVe is the CSMS the 47 scenarios were originally written - # against, so it is where a capability the harness LOST shows up first. Its - # scope table claims every scenario; a row that had to be demoted would mean - # the core dropped something. Without this job that can rot green unnoticed. - e2e-steve: + # SteVe is the CSMS the 47 scenarios were originally written against, so it + # is where a capability the harness LOST shows up first. Its scope table + # claims every scenario; a row that had to be demoted would mean the core + # dropped something. Without this lane that can rot green unnoticed. + # + # CitrineOS had no part in writing the scenarios, so it is where the core + # would be caught quietly re-acquiring a dependency on how SteVe happens to + # behave. A CSMS-neutrality claim that only one CSMS ever exercises is a + # claim, not a check. It carries one known finding, and carries it in the + # open: cert16-tc023-3-authorize-blocked fails deterministically (3 runs of + # 3) because CitrineOS answers Invalid where OCPP 1.6 requires Blocked. The + # row is kept red rather than demoted in the scope table, which tck/scope.ts + # forbids, and it is DECLARED in drivers/citrineos/expected.ts -- so the + # sweep reports it as EXPECTED FAIL and exits 0, a NEW red still fails the + # build, and the day upstream fixes the mapping that row fails as UNEXPECTED + # PASS until its entry is deleted. drivers/citrineos/README.md carries the + # full gap table. + # + # ONE JOB RATHER THAN TWO NEAR-IDENTICAL COPIES, and it is only legal because + # both lanes are blocking: a matrix carries one continue-on-error for the + # whole strategy, so the arrangement this replaced -- one lane muted, one not + # -- could not have been written here at all. + e2e: runs-on: ubuntu-latest needs: check - # ~10 min of sweep plus image pull, a cold SteVe boot and the one-off - # provisioning restart. Generous, but far below the 6h default: a wedged - # simulator container should fail the build, not occupy a runner all day. + strategy: + # NOT the default, and not a preference. fail-fast cancels the sibling on + # the first failure, which would destroy the exact property the two + # paragraphs above describe: each lane answers a question the other + # cannot, and a cancelled lane is a missing answer rather than a saved + # minute. + fail-fast: false + matrix: + driver: [steve, citrineos] + # ~10 min of sweep plus image pull, a cold CSMS boot and -- for SteVe -- + # the one-off provisioning restart. Generous, but far below the 6h default: + # a wedged simulator container should fail the build, not occupy a runner + # all day. timeout-minutes: 45 env: - CSMS_DRIVER: ./drivers/steve/index.ts - # localhost, not the compose service name: the runner reaches SteVe + CSMS_DRIVER: ./drivers/${{ matrix.driver }}/index.ts + OCPP_CP_IDS: CERTCP1,CERTCP2,CERTCP3 + # localhost, not the compose service name: the runner reaches the CSMS # through the published port, while the simulator containers reach it by - # service name from inside the network. + # service name from inside the network. That asymmetry is why each driver + # has a separate setting for the two. + # + # Both are set in both lanes. Each driver reads only its own, so the + # unused one is inert, and spelling them out here beats a + # matrix.include that hides which settings exist at all. STEVE_URL: http://localhost:8180/steve/manager - OCPP_CP_IDS: CERTCP1,CERTCP2,CERTCP3 + CITRINE_API_URL: http://localhost:8080 steps: - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 with: - # Neither job pushes anything, so the token has no reason to stay - # in .git/config where every later step -- install scripts, the - # sweep, the containers it spawns -- could read it. + # This job pushes nothing, so the token has no reason to stay in + # .git/config where every later step -- install scripts, the sweep, + # the containers it spawns -- could read it. persist-credentials: false # Started before the toolchain, and deliberately without --wait: pulling - # two pinned images and cold-booting SteVe (its healthcheck allows 150s - # for Flyway) shares nothing with installing bun, so the two may as well - # overlap. The blocking wait below is then usually already satisfied. - - name: Start SteVe - run: docker compose -f drivers/steve/compose.yaml up -d + # the pinned images and cold-booting the CSMS (SteVe's healthcheck allows + # 150s for Flyway; CitrineOS runs its migrations on a cold database) + # shares nothing with installing bun, so the two may as well overlap. The + # blocking wait below is then usually already satisfied. + - name: Start the CSMS + run: docker compose -f drivers/${{ matrix.driver }}/compose.yaml up -d - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 with: @@ -123,7 +159,7 @@ jobs: - run: bun install --frozen-lockfile - name: Wait for it to be healthy - run: docker compose -f drivers/steve/compose.yaml up -d --wait + run: docker compose -f drivers/${{ matrix.driver }}/compose.yaml up -d --wait - name: Seed the fixtures the scenarios assume run: bun bin/ocpp-tck.ts driver provision @@ -143,20 +179,22 @@ jobs: # barely moves with runner speed. Across 3 lanes, core is ~3 min and all # 44 are ~10 min. Roughly 7 minutes buys back the 29 scenarios that core # leaves out -- on precisely the branch where a regression is introduced, - # and for a job whose entire reason to exist is that this driver cannot + # and for a job whose entire reason to exist is that these drivers cannot # rot green. Cheap coverage is not worth a stale signal. # # --retry-failed-isolated is not optional here: the runner warns that # parallel lanes are not fully isolated, so a bare --parallel turns # cross-lane interference into a red build. The retry re-runs only the # failures, sequentially, which is the mode the runner calls reliable. - # `bun run e2e`, not the two commands spelled out here. A full sweep is - # two invocations -- `all` is 44 scenarios and the authorize group - # (TC_023) sits outside it, see TODO(upstream) at the GROUPS registry in + # + # `bun run e2e`, not the two commands it wraps. A full sweep is two + # invocations -- `all` is 44 scenarios and the authorize group (TC_023) + # sits outside it, see TODO(upstream) at the GROUPS registry in # tck/main.ts -- and that fact used to live only in this file. Anyone # running `run-all` by hand got 44/47 and read "no failures", missing # exactly the three scenarios that prove `driver provision` seeded - # anything. One script means CI and a human run the same thing. + # anything. One script means CI and a human run the same thing, and it + # means the two lanes cannot drift apart either. - name: Conformance sweep run: bun run e2e @@ -165,92 +203,10 @@ jobs: if: always() uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: - name: ocpp-tck-steve-results-${{ github.run_id }} - path: results/ - if-no-files-found: warn - - - name: Tear down - if: always() - run: docker compose -f drivers/steve/compose.yaml down -v - - # The peer of the job above, answering the question it cannot: CitrineOS had - # no part in writing the scenarios, so this is where the core would be caught - # quietly re-acquiring a dependency on how SteVe happens to behave. A - # CSMS-neutrality claim that only one CSMS ever exercises is a claim, not a - # check. - # - # BLOCKING, while still carrying a known finding. One of the 47 scenarios - # fails against the pinned CitrineOS -- cert16-tc023-3-authorize-blocked, - # deterministic (3 runs of 3): CitrineOS answers Invalid where OCPP 1.6 - # requires Blocked. It is kept red rather than demoted in the scope table, - # which tck/scope.ts forbids, and it is DECLARED in - # drivers/citrineos/expected.ts, so the sweep reports it as EXPECTED FAIL and - # exits 0. - # - # This job used to be continue-on-error for that one row, and that is what - # the declaration replaced. continue-on-error is job-granular where the - # intent is scenario-granular: a broken compose.yaml, a failed provision, a - # verify mismatch and a regression from 1 failure to 20 were all as green as - # the finding it was muted for, so the job could not report the one thing it - # exists to report -- the other 46. Now a new red fails the build, and the - # day upstream fixes the Blocked mapping that row fails as UNEXPECTED PASS - # until its entry is deleted. drivers/citrineos/README.md carries the full - # gap table. - e2e-citrineos: - runs-on: ubuntu-latest - needs: check - timeout-minutes: 45 - env: - CSMS_DRIVER: ./drivers/citrineos/index.ts - # localhost, not the compose service name -- same asymmetry as the SteVe - # job: the runner reaches the message API through the published port, - # while the simulator containers reach the OCPP endpoint by service name - # from inside the network. - CITRINE_API_URL: http://localhost:8080 - OCPP_CP_IDS: CERTCP1,CERTCP2,CERTCP3 - steps: - - uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 - with: - persist-credentials: false - - # Before the toolchain and without --wait, for the same reason as the - # SteVe job: pulling the images and running the migrations on a cold - # database shares nothing with installing bun. - - name: Start CitrineOS - run: docker compose -f drivers/citrineos/compose.yaml up -d - - - uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0 - with: - bun-version: ${{ env.BUN_VERSION }} - - - run: bun install --frozen-lockfile - - - name: Wait for it to be healthy - run: docker compose -f drivers/citrineos/compose.yaml up -d --wait - - - name: Seed the fixtures the scenarios assume - run: bun bin/ocpp-tck.ts driver provision - - # Same reasoning as the SteVe job: a verb nothing runs is a verb that rots. - - name: Environment matches the scenarios - run: bun bin/ocpp-tck.ts driver verify - - # `bun run e2e`, exactly as the SteVe job runs it -- both sweeps, one - # script, so the two jobs cannot drift apart and neither can drift from - # what a human runs by hand. Seven scenarios are NOT APPLICABLE here and - # never start a container, so this is cheaper than SteVe's rather than - # doubling the bill. - - name: Conformance sweep - run: bun run e2e - - - name: Upload results - if: always() - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 - with: - name: ocpp-tck-citrineos-results-${{ github.run_id }} + name: ocpp-tck-${{ matrix.driver }}-results-${{ github.run_id }} path: results/ if-no-files-found: warn - name: Tear down if: always() - run: docker compose -f drivers/citrineos/compose.yaml down -v + run: docker compose -f drivers/${{ matrix.driver }}/compose.yaml down -v From 41c2c9e649634399018a74633ace5181e683d89d Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 14 Aug 2026 10:39:54 +0200 Subject: [PATCH 05/11] fix(runner): say why an expected failure stopped failing Review of the previous commits. The UNEXPECTED PASS message implies deleting a recorded finding, so what it says has to be right; three cases were wrong or inconsistent. PARTIAL was reported as "the finding looks fixed". It is the opposite: PARTIAL means a check degraded to SKIPPED because the driver could not evaluate it, so if the degraded check is the one that caught the finding, there is no evidence either way. Still red -- an unmeasurable declared row deserves attention -- but the message now says the evidence is missing rather than that the CSMS was fixed. NOT APPLICABLE was diagnosed twice, differently. The sweep called it a contradiction between the scope table and the list; the single-scenario path told the reader to delete the entry, which may well be the wrong half. unexpectedPassDetail() now takes the fields instead of a ScenarioOutcome, so both paths reach the same wording. The comment claiming NOT APPLICABLE was unreachable in standingOf() was false, and contradicted the branch immediately below it: check-driver only rejects the contradiction for a driver that has been checked, and the runtime UnsupportedOperationError escape produces it after any offline check could see it. Also: `check-driver --json` emits the entries rather than their ids. That output is what a conformance report reads, and "which scenarios are excused, on what recorded evidence" is a question it should not have to open the driver's source to answer. And a note recording that a THROW out of a single `ocpp-tck run` stays fatal however the scenario is declared, where the sweep would catch it as ERROR and excuse it -- deliberate, not an oversight: that path is the debugging mode. --- VENDOR.md | 2 +- patches/tck/main.ts.patch | 150 +++++++++++++++++++++++++------------- tck/main.ts | 106 +++++++++++++++++++-------- 3 files changed, 177 insertions(+), 81 deletions(-) diff --git a/VENDOR.md b/VENDOR.md index a5719fd..6270419 100644 --- a/VENDOR.md +++ b/VENDOR.md @@ -57,7 +57,7 @@ strongest available demonstration that the core names no CSMS. | `tck/specs/index.ts` | `upstream-verbatim` | `scripts/steve-verify/runner/specs/index.ts` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `—` | | `tck/assert.ts` | `upstream-patched` | `src/cp/application/verification/assert.ts` | `2431f5f6c0df997d4d821d9af55689c1f0f2df199de1e9e4ed6f3fbaad4fc89e` | `76d0f293db4f6ecc5768affea0ca76d2841c147fa3687dbea1f9e950bfbd9298` | `patches/tck/assert.ts.patch` | | `tck/sim.ts` | `upstream-patched` | `scripts/steve-verify/runner/sim.ts` | `2bf2f78afe3434e7139cd62c3ff6d70f02defd39dd700611e7c5f7614260cd35` | `45f897f273ea73227bb9b30766127eb9ecae0a319ab9048cbc30428fafff7f32` | `patches/tck/sim.ts.patch` | -| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `457981bdfd6bb42253acc62fd76ca49043bd2e4e47c33476a4415a1dd7414040` | `patches/tck/main.ts.patch` | +| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `4ffd8ca1a6df9ff59b7514f2a6546348255587ac3d9c0764c4415e20d2a0d04c` | `patches/tck/main.ts.patch` | | `tck/driver.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/index.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/driver-registry.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | diff --git a/patches/tck/main.ts.patch b/patches/tck/main.ts.patch index 6736b32..193b41a 100644 --- a/patches/tck/main.ts.patch +++ b/patches/tck/main.ts.patch @@ -222,8 +222,8 @@ + templateId: string, +): Promise { + return expectedFailureFor(await expectedFailureTable(), templateId); -+} -+ + } + +/** Result of one scenario execution -- either it produced checks, or the + * driver declared the scenario undrivable partway through drive(). */ +type ScenarioRun = @@ -261,8 +261,8 @@ +async function driverModule(): Promise { + driverModulePromise ??= loadDriverModule(); + return driverModulePromise; - } - ++} ++ +async function driver(): Promise { + driverPartsPromise ??= (async () => { + const module = await driverModule(); @@ -581,7 +581,7 @@ all: [ ...CORE_SPECS, ...AUTHLIST_RESERVATION_SPECS, -@@ -286,66 +523,191 @@ +@@ -286,66 +523,219 @@ .map((spec) => [spec.templateId, spec]), ); @@ -688,7 +688,7 @@ + * + * A parallel-lane FAIL/ERROR that did not fail its isolated retry is a flake, + * not a failure; see {@link retryFailedOutcomesIsolated}. - */ ++ */ +function effectivelyFailed(outcome: ScenarioOutcome): boolean { + if (!isFailure(outcome.verdict)) return false; + return ( @@ -723,29 +723,57 @@ + return failed ? "unexpected-fail" : "ok"; + } + if (failed) return "expected-fail"; -+ // NOT APPLICABLE is unreachable here: check-driver rejects an id that is -+ // both declared expected-failing and NOT_APPLICABLE in the scope table, and -+ // the runtime UnsupportedOperationError escape would be the same -+ // contradiction arriving late. Reported as an unexpected pass rather than -+ // silently tolerated -- "it never ran" is not "it was fixed", and the -+ // message below says which. ++ // Everything else is an unexpected pass, INCLUDING NOT APPLICABLE. That one ++ // is not merely theoretical: check-driver rejects an id declared both ways, ++ // but only a driver that has been checked, and the runtime ++ // UnsupportedOperationError escape produces the same contradiction after the ++ // fact, when no offline check can have seen it. It is surfaced rather than ++ // tolerated because "it never ran" is not "it was fixed" -- ++ // unexpectedPassDetail() is where the difference gets said. + return "unexpected-pass"; +} + -+/** Why an unexpected pass is one, in the words a reader needs to tell a fix -+ * from a flake from a contradiction without re-reading this file. */ -+function unexpectedPassDetail(outcome: ScenarioOutcome): string { -+ if (outcome.verdict === "NOT APPLICABLE") { -+ return "it never ran (NOT APPLICABLE), so it cannot be failing as declared"; ++/** ++ * Why an unexpected pass is one, in the words a reader needs to tell a fix ++ * from a flake from a contradiction from an unmeasured run -- without ++ * re-reading this file. ++ * ++ * The wording carries real weight, because the action it implies is DELETING A ++ * RECORDED FINDING. Only one of the four cases below is actually evidence that ++ * the CSMS was fixed; saying so for the other three would talk a maintainer ++ * into throwing away a finding that still holds. ++ * ++ * Takes the fields rather than the outcome, so that the single-scenario path ++ * -- which has no ScenarioOutcome and no isolated retry -- diagnoses the same ++ * situation the same way. It used to have a message of its own, and the two ++ * disagreed about the NOT APPLICABLE case. ++ */ ++function unexpectedPassDetail( ++ verdict: Verdict, ++ isolatedRetry?: RetryOutcome, ++): string { ++ if (verdict === "NOT APPLICABLE") { ++ return ( ++ "it never ran (NOT APPLICABLE) -- the scope table and the " + ++ "expected-failure list contradict each other, so the contradiction is " + ++ "what to fix, and deleting the entry may be the wrong half" ++ ); + } -+ if (isFailure(outcome.verdict)) { ++ if (verdict === "PARTIAL") { + return ( -+ `it failed in the parallel lane and ${outcome.isolatedRetry?.verdict} ` + -+ "on its isolated retry -- either upstream fixed it, or the scenario is " + -+ "flaky and the flake is the bug to fix" ++ "it came back PARTIAL -- at least one check was SKIPPED because the " + ++ "driver could not evaluate it, so this is NOT evidence that the finding " + ++ "is fixed; find out which check degraded before deleting the entry" + ); + } -+ return `it came back ${outcome.verdict} outright -- the finding looks fixed`; ++ if (isolatedRetry !== undefined && isFailure(verdict)) { ++ return ( ++ `it failed in the parallel lane and ${isolatedRetry.verdict} on its ` + ++ "isolated retry -- either upstream fixed it, or the scenario is flaky " + ++ "and the flake is the bug to fix" ++ ); ++ } ++ return `it came back ${verdict} outright -- the finding looks fixed`; +} + +/** @@ -755,7 +783,7 @@ + * + * The scope table is consulted FIRST -- a NOT_APPLICABLE scenario never + * starts a container. -+ */ + */ async function runOneForSweep( spec: ScenarioSpec, cpId: string, @@ -811,7 +839,7 @@ }; } catch (err) { const message = -@@ -359,58 +721,47 @@ +@@ -359,58 +749,47 @@ verdict: "ERROR", checks: null, failed: null, @@ -892,7 +920,7 @@ ); for (const outcome of toRetry) { -@@ -432,13 +783,14 @@ +@@ -432,13 +811,14 @@ verdict: retryOutcome.verdict, checks: retryOutcome.checks, failed: retryOutcome.failed, @@ -909,7 +937,7 @@ : " -- CONFIRMED (fails isolated too, not a parallel-lane artifact)\n"), ); } -@@ -451,11 +803,43 @@ +@@ -451,11 +831,43 @@ return new Date().toISOString().replace(/\.\d{3}Z$/, "Z"); } @@ -958,7 +986,7 @@ async function writeSummary( groupName: string, outcomes: ScenarioOutcome[], -@@ -465,45 +849,97 @@ +@@ -465,45 +877,97 @@ const rows = outcomes.map((o) => { const checks = o.checks === null ? "-" : String(o.checks); const failed = o.failed === null ? "-" : String(o.failed); @@ -975,7 +1003,7 @@ + } else if (standing === "unexpected-pass") { + verdict = + `${verdict} — UNEXPECTED PASS: declared expected-failing, but ` + -+ `${unexpectedPassDetail(o)}. Delete the entry or re-word it.`; ++ `${unexpectedPassDetail(o.verdict, o.isolatedRetry)}.`; + } + const base = `| ${o.templateId} | ${o.cpId} | ${verdict} | ${checks} | ${failed} | ${skipped} |`; if (!anyRetried) return base; @@ -1035,7 +1063,7 @@ + "known-red entry gets deleted when it stops being true, instead of " + + "outliving the defect it documents.", + ...unexpectedPasses.map( -+ (o) => `- \`${o.templateId}\` — ${unexpectedPassDetail(o)}`, ++ (o) => `- \`${o.templateId}\` — ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}`, + ), + ); + } @@ -1071,7 +1099,7 @@ "", header, separator, -@@ -518,54 +954,53 @@ +@@ -518,54 +982,53 @@ } /** Sequential or parallel group sweep. Writes results/summary.md and exits @@ -1151,7 +1179,7 @@ ); } else { for (let i = 0; i < specs.length; i++) { -@@ -574,11 +1009,11 @@ +@@ -574,11 +1037,11 @@ } if (retryFailedIsolated) { @@ -1165,7 +1193,7 @@ "(a sequential sweep is already isolated).\n", ); } -@@ -588,45 +1023,85 @@ +@@ -588,45 +1051,85 @@ for (const o of outcomes) { const retrySuffix = o.isolatedRetry ? ` [isolated retry: ${o.isolatedRetry.verdict}${ @@ -1179,7 +1207,7 @@ + standing === "expected-fail" + ? ` [EXPECTED FAIL: ${o.expected?.reason} (${o.expected?.finding})]` + : standing === "unexpected-pass" -+ ? ` [UNEXPECTED PASS: declared expected-failing, but ${unexpectedPassDetail(o)}]` ++ ? ` [UNEXPECTED PASS: declared expected-failing, but ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}]` + : ""; process.stderr.write( - ` ${o.verdict}: ${o.templateId} (${o.cpId}, ${o.checks ?? "-"} checks, ${o.failed ?? "-"} failed)${retrySuffix}\n`, @@ -1234,7 +1262,7 @@ + ); + for (const o of unexpectedPasses) { + process.stderr.write( -+ ` ${o.templateId}: ${unexpectedPassDetail(o)} -- delete its expected-failure entry, or re-word it to say what is still true.\n`, ++ ` ${o.templateId}: ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}.\n`, + ); + } + } @@ -1266,7 +1294,7 @@ } // --------------------------------------------------------------------------- -@@ -642,6 +1117,8 @@ +@@ -642,6 +1145,8 @@ cpId: string; connector?: number; timeoutSecs?: number; @@ -1275,7 +1303,7 @@ } function requireValue(argv: string[], index: number, flag: string): string { -@@ -663,37 +1140,80 @@ +@@ -663,37 +1168,80 @@ return n; } @@ -1373,7 +1401,7 @@ if (argv[0] === "run-all") { group = "all"; -@@ -708,18 +1228,27 @@ +@@ -708,18 +1256,27 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1404,7 +1432,7 @@ process.exit(1); } -@@ -749,9 +1278,11 @@ +@@ -749,9 +1306,11 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1417,7 +1445,7 @@ process.exit(1); } } -@@ -764,35 +1295,486 @@ +@@ -764,35 +1323,495 @@ cpId, connector, timeoutSecs, @@ -1589,7 +1617,7 @@ + ); + } + } - ++ + // Everything about the expected-failure list that is knowable offline. What + // is NOT knowable here is whether those scenarios still fail -- only a sweep + // answers that, by reporting an UNEXPECTED PASS. @@ -1629,7 +1657,7 @@ + ); + } + } -+ + + if (capabilities) { + const known = new Set(CSMS_OPERATION_ACTIONS); + const unknown = [...capabilities.operations].filter( @@ -1664,7 +1692,16 @@ + NOT_APPLICABLE: templateIdsWithStatus(scope, "NOT_APPLICABLE").length, + } + : null, -+ expectedFailures: expected ? Object.keys(expected).sort() : null, ++ // The entries, not their count and not their ids: --json is what a ++ // conformance report would read, and "which scenarios are excused, and on ++ // what recorded evidence" is the question it has to answer without going ++ // back to the driver's source. Sorted, so two runs of the same driver ++ // diff cleanly. ++ expectedFailures: expected ++ ? Object.keys(expected) ++ .sort() ++ .map((templateId) => ({ templateId, ...expected[templateId] })) ++ : null, + problems, + warnings, + }; @@ -1702,8 +1739,8 @@ + "-- each still runs, still prints FAIL, and fails the sweep the day it " + + "passes:\n", + ); -+ for (const id of summary.expectedFailures) { -+ process.stderr.write(` ${id}: ${expected?.[id]?.finding}\n`); ++ for (const entry of summary.expectedFailures) { ++ process.stderr.write(` ${entry.templateId}: ${entry.finding}\n`); + } + } + return 0; @@ -1920,7 +1957,7 @@ } const spec = SPECS_BY_TEMPLATE_ID.get(args.templateId); -@@ -800,18 +1782,70 @@ +@@ -800,18 +1819,81 @@ process.stderr.write( `Unknown template id: ${args.templateId} (known: ${[...SPECS_BY_TEMPLATE_ID.keys()].join(", ")})\n`, ); @@ -1975,6 +2012,13 @@ + } + return 1; + } ++ // Only an ASSERTION failure is excusable here. A throw out of runScenario -- ++ // a bounded wait giving up, a driver blowing up -- propagates to ++ // bin/ocpp-tck.ts and exits 1 whatever the declaration says, where the sweep ++ // would have caught it as ERROR and excused it. The asymmetry is deliberate: ++ // this is the debugging mode, a stack trace is the thing worth having, and a ++ // declared finding about what a CSMS ANSWERS is not a licence to swallow a ++ // crash. CI runs the sweep, so no signal depends on the difference. + return unexpectedPassExit(spec.templateId, verdict, expected); } + @@ -1982,9 +2026,14 @@ + * Exit code for a single scenario that did NOT fail: 0 normally, 1 when the + * driver declared it expected-failing. + * -+ * A single run has no isolated retry, so there is no flake to distinguish -+ * here -- one non-failure IS the whole evidence, and the message says so -+ * rather than asserting the finding is fixed. ++ * Shares unexpectedPassDetail() with the sweep rather than wording its own ++ * diagnosis. It did word its own, and the two disagreed on the case that ++ * matters most: a NOT APPLICABLE row that is also declared expected-failing is ++ * a contradiction between two tables, and this path told the reader to delete ++ * the entry while the sweep told them the contradiction was the thing to fix. ++ * ++ * A single run has no isolated retry, so there is no flake to distinguish -- ++ * one non-failure IS the whole evidence here, and the detail says which kind. + */ +function unexpectedPassExit( + templateId: string, @@ -1994,9 +2043,8 @@ + if (!expected) return 0; + process.stderr.write( + `[runner] ${templateId} UNEXPECTED PASS: this driver declares it ` + -+ `expected-failing (${expected.reason} -- ${expected.finding}), and it ` + -+ `came back ${verdict}. Delete the expected-failure entry, or re-word it ` + -+ "to say what is still true.\n", ++ `expected-failing (${expected.reason} -- ${expected.finding}), and ` + ++ `${unexpectedPassDetail(verdict)}.\n`, + ); + return 1; +} diff --git a/tck/main.ts b/tck/main.ts index 3d6b015..b744ecc 100644 --- a/tck/main.ts +++ b/tck/main.ts @@ -627,29 +627,57 @@ function standingOf(outcome: ScenarioOutcome): SweepStanding { return failed ? "unexpected-fail" : "ok"; } if (failed) return "expected-fail"; - // NOT APPLICABLE is unreachable here: check-driver rejects an id that is - // both declared expected-failing and NOT_APPLICABLE in the scope table, and - // the runtime UnsupportedOperationError escape would be the same - // contradiction arriving late. Reported as an unexpected pass rather than - // silently tolerated -- "it never ran" is not "it was fixed", and the - // message below says which. + // Everything else is an unexpected pass, INCLUDING NOT APPLICABLE. That one + // is not merely theoretical: check-driver rejects an id declared both ways, + // but only a driver that has been checked, and the runtime + // UnsupportedOperationError escape produces the same contradiction after the + // fact, when no offline check can have seen it. It is surfaced rather than + // tolerated because "it never ran" is not "it was fixed" -- + // unexpectedPassDetail() is where the difference gets said. return "unexpected-pass"; } -/** Why an unexpected pass is one, in the words a reader needs to tell a fix - * from a flake from a contradiction without re-reading this file. */ -function unexpectedPassDetail(outcome: ScenarioOutcome): string { - if (outcome.verdict === "NOT APPLICABLE") { - return "it never ran (NOT APPLICABLE), so it cannot be failing as declared"; +/** + * Why an unexpected pass is one, in the words a reader needs to tell a fix + * from a flake from a contradiction from an unmeasured run -- without + * re-reading this file. + * + * The wording carries real weight, because the action it implies is DELETING A + * RECORDED FINDING. Only one of the four cases below is actually evidence that + * the CSMS was fixed; saying so for the other three would talk a maintainer + * into throwing away a finding that still holds. + * + * Takes the fields rather than the outcome, so that the single-scenario path + * -- which has no ScenarioOutcome and no isolated retry -- diagnoses the same + * situation the same way. It used to have a message of its own, and the two + * disagreed about the NOT APPLICABLE case. + */ +function unexpectedPassDetail( + verdict: Verdict, + isolatedRetry?: RetryOutcome, +): string { + if (verdict === "NOT APPLICABLE") { + return ( + "it never ran (NOT APPLICABLE) -- the scope table and the " + + "expected-failure list contradict each other, so the contradiction is " + + "what to fix, and deleting the entry may be the wrong half" + ); + } + if (verdict === "PARTIAL") { + return ( + "it came back PARTIAL -- at least one check was SKIPPED because the " + + "driver could not evaluate it, so this is NOT evidence that the finding " + + "is fixed; find out which check degraded before deleting the entry" + ); } - if (isFailure(outcome.verdict)) { + if (isolatedRetry !== undefined && isFailure(verdict)) { return ( - `it failed in the parallel lane and ${outcome.isolatedRetry?.verdict} ` + - "on its isolated retry -- either upstream fixed it, or the scenario is " + - "flaky and the flake is the bug to fix" + `it failed in the parallel lane and ${isolatedRetry.verdict} on its ` + + "isolated retry -- either upstream fixed it, or the scenario is flaky " + + "and the flake is the bug to fix" ); } - return `it came back ${outcome.verdict} outright -- the finding looks fixed`; + return `it came back ${verdict} outright -- the finding looks fixed`; } /** @@ -861,7 +889,7 @@ async function writeSummary( } else if (standing === "unexpected-pass") { verdict = `${verdict} — UNEXPECTED PASS: declared expected-failing, but ` + - `${unexpectedPassDetail(o)}. Delete the entry or re-word it.`; + `${unexpectedPassDetail(o.verdict, o.isolatedRetry)}.`; } const base = `| ${o.templateId} | ${o.cpId} | ${verdict} | ${checks} | ${failed} | ${skipped} |`; if (!anyRetried) return base; @@ -916,7 +944,7 @@ async function writeSummary( "known-red entry gets deleted when it stops being true, instead of " + "outliving the defect it documents.", ...unexpectedPasses.map( - (o) => `- \`${o.templateId}\` — ${unexpectedPassDetail(o)}`, + (o) => `- \`${o.templateId}\` — ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}`, ), ); } @@ -1032,7 +1060,7 @@ async function runGroupSweep( standing === "expected-fail" ? ` [EXPECTED FAIL: ${o.expected?.reason} (${o.expected?.finding})]` : standing === "unexpected-pass" - ? ` [UNEXPECTED PASS: declared expected-failing, but ${unexpectedPassDetail(o)}]` + ? ` [UNEXPECTED PASS: declared expected-failing, but ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}]` : ""; process.stderr.write( ` ${o.verdict}: ${o.templateId} (${o.cpId}, ${o.checks ?? "-"} checks, ${o.failed ?? "-"} failed, ${o.skipped ?? "-"} skipped)${retrySuffix}${declaredSuffix}${reasonSuffix}\n`, @@ -1079,7 +1107,7 @@ async function runGroupSweep( ); for (const o of unexpectedPasses) { process.stderr.write( - ` ${o.templateId}: ${unexpectedPassDetail(o)} -- delete its expected-failure entry, or re-word it to say what is still true.\n`, + ` ${o.templateId}: ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}.\n`, ); } } @@ -1532,7 +1560,16 @@ async function checkDriver(argv: string[]): Promise { NOT_APPLICABLE: templateIdsWithStatus(scope, "NOT_APPLICABLE").length, } : null, - expectedFailures: expected ? Object.keys(expected).sort() : null, + // The entries, not their count and not their ids: --json is what a + // conformance report would read, and "which scenarios are excused, and on + // what recorded evidence" is the question it has to answer without going + // back to the driver's source. Sorted, so two runs of the same driver + // diff cleanly. + expectedFailures: expected + ? Object.keys(expected) + .sort() + .map((templateId) => ({ templateId, ...expected[templateId] })) + : null, problems, warnings, }; @@ -1570,8 +1607,8 @@ async function checkDriver(argv: string[]): Promise { "-- each still runs, still prints FAIL, and fails the sweep the day it " + "passes:\n", ); - for (const id of summary.expectedFailures) { - process.stderr.write(` ${id}: ${expected?.[id]?.finding}\n`); + for (const entry of summary.expectedFailures) { + process.stderr.write(` ${entry.templateId}: ${entry.finding}\n`); } } return 0; @@ -1824,6 +1861,13 @@ export async function cli(argv: string[]): Promise { } return 1; } + // Only an ASSERTION failure is excusable here. A throw out of runScenario -- + // a bounded wait giving up, a driver blowing up -- propagates to + // bin/ocpp-tck.ts and exits 1 whatever the declaration says, where the sweep + // would have caught it as ERROR and excused it. The asymmetry is deliberate: + // this is the debugging mode, a stack trace is the thing worth having, and a + // declared finding about what a CSMS ANSWERS is not a licence to swallow a + // crash. CI runs the sweep, so no signal depends on the difference. return unexpectedPassExit(spec.templateId, verdict, expected); } @@ -1831,9 +1875,14 @@ export async function cli(argv: string[]): Promise { * Exit code for a single scenario that did NOT fail: 0 normally, 1 when the * driver declared it expected-failing. * - * A single run has no isolated retry, so there is no flake to distinguish - * here -- one non-failure IS the whole evidence, and the message says so - * rather than asserting the finding is fixed. + * Shares unexpectedPassDetail() with the sweep rather than wording its own + * diagnosis. It did word its own, and the two disagreed on the case that + * matters most: a NOT APPLICABLE row that is also declared expected-failing is + * a contradiction between two tables, and this path told the reader to delete + * the entry while the sweep told them the contradiction was the thing to fix. + * + * A single run has no isolated retry, so there is no flake to distinguish -- + * one non-failure IS the whole evidence here, and the detail says which kind. */ function unexpectedPassExit( templateId: string, @@ -1843,9 +1892,8 @@ function unexpectedPassExit( if (!expected) return 0; process.stderr.write( `[runner] ${templateId} UNEXPECTED PASS: this driver declares it ` + - `expected-failing (${expected.reason} -- ${expected.finding}), and it ` + - `came back ${verdict}. Delete the expected-failure entry, or re-word it ` + - "to say what is still true.\n", + `expected-failing (${expected.reason} -- ${expected.finding}), and ` + + `${unexpectedPassDetail(verdict)}.\n`, ); return 1; } From 503f0d3424b21424505fd79d39d809dec19453bf Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 14 Aug 2026 11:03:35 +0200 Subject: [PATCH 06/11] refactor(runner): one classifier, one declaration sentence Cleanup pass over the expected-failure commits. No behaviour change: the sweep, the summary and the single-scenario path produce the same output and the same exit codes as before, checked on all four paths. standingOf() took a ScenarioOutcome, which the single-scenario path does not have, so that path re-implemented the rule in nested ifs. It now takes the fields, with standingOfOutcome() as the sweep's one-line wrapper and exitForSingleRun() switching on the result. This is the same reshape unexpectedPassDetail() already got, and for the same reason: the two paths had disagreed once about NOT APPLICABLE, and agreeing today is weaker than being unable to disagree. The "EXPECTED FAIL: reason (finding)" sentence was rebuilt at three call sites and a fourth copy had drifted to a different separator. declarationNote() owns it, which also lets the callers narrow `expected` -- they were reaching through `?.` and would have rendered the literal "undefined" into a report if the invariant ever broke. flakeCount still spelled out the predicate by hand, directly under a comment claiming everything below read it through effectivelyFailed(). It does now. runOneForSweep()'s four returns repeated the same fields; the shared half is hoisted, which makes "the declaration is carried on every branch" structural rather than four sites each remembering. Also corrected isFailure()'s docstring, which still claimed those two verdicts decide the exit code. REJECTED, and recorded on EnvDependent so it is not proposed again: folding driverScope/driverCapabilities/driverExpectedFailures onto one generic resolver. Over an unconstrained T the union becomes `((env) => T) | (T & Function)`, `typeof` stops narrowing it and the helper needs a cast -- losing the cast-free narrowing this type is documented to provide, for a rule whose whole failure mode is a declaration being read as its own resolver. --- VENDOR.md | 2 +- patches/tck/main.ts.patch | 300 +++++++++++++++++++++----------------- tck/driver.ts | 11 +- tck/main.ts | 234 +++++++++++++++-------------- 4 files changed, 296 insertions(+), 251 deletions(-) diff --git a/VENDOR.md b/VENDOR.md index 6270419..4cdade1 100644 --- a/VENDOR.md +++ b/VENDOR.md @@ -57,7 +57,7 @@ strongest available demonstration that the core names no CSMS. | `tck/specs/index.ts` | `upstream-verbatim` | `scripts/steve-verify/runner/specs/index.ts` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `—` | | `tck/assert.ts` | `upstream-patched` | `src/cp/application/verification/assert.ts` | `2431f5f6c0df997d4d821d9af55689c1f0f2df199de1e9e4ed6f3fbaad4fc89e` | `76d0f293db4f6ecc5768affea0ca76d2841c147fa3687dbea1f9e950bfbd9298` | `patches/tck/assert.ts.patch` | | `tck/sim.ts` | `upstream-patched` | `scripts/steve-verify/runner/sim.ts` | `2bf2f78afe3434e7139cd62c3ff6d70f02defd39dd700611e7c5f7614260cd35` | `45f897f273ea73227bb9b30766127eb9ecae0a319ab9048cbc30428fafff7f32` | `patches/tck/sim.ts.patch` | -| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `4ffd8ca1a6df9ff59b7514f2a6546348255587ac3d9c0764c4415e20d2a0d04c` | `patches/tck/main.ts.patch` | +| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `7a0e8b6a19a13a8ce80a426ba63b77b8d23715d78f2ddd8dc4e7a6943fea5f15` | `patches/tck/main.ts.patch` | | `tck/driver.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/index.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/driver-registry.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | diff --git a/patches/tck/main.ts.patch b/patches/tck/main.ts.patch index 193b41a..7c93359 100644 --- a/patches/tck/main.ts.patch +++ b/patches/tck/main.ts.patch @@ -139,14 +139,16 @@ // --------------------------------------------------------------------------- // Runner core // --------------------------------------------------------------------------- -@@ -60,95 +114,242 @@ +@@ -60,95 +114,244 @@ timeoutSecs?: number; } +/** Verdict for one scenario. PASS/FAIL/ERROR keep their upstream meaning. */ +type Verdict = "PASS" | "PARTIAL" | "FAIL" | "ERROR" | "NOT APPLICABLE"; + -+/** Only these two make the process exit non-zero. */ ++/** The two verdicts that count as a failure. Whether one ends the process ++ * non-zero is {@link standingOf}'s answer, not this one's -- see the exit-code ++ * rule at the top of this file. */ +function isFailure(verdict: Verdict): boolean { + return verdict === "FAIL" || verdict === "ERROR"; +} @@ -222,8 +224,8 @@ + templateId: string, +): Promise { + return expectedFailureFor(await expectedFailureTable(), templateId); - } - ++} ++ +/** Result of one scenario execution -- either it produced checks, or the + * driver declared the scenario undrivable partway through drive(). */ +type ScenarioRun = @@ -261,8 +263,8 @@ +async function driverModule(): Promise { + driverModulePromise ??= loadDriverModule(); + return driverModulePromise; -+} -+ + } + +async function driver(): Promise { + driverPartsPromise ??= (async () => { + const module = await driverModule(); @@ -443,7 +445,7 @@ 30_000, ); } catch (err) { -@@ -176,13 +377,25 @@ +@@ -176,13 +379,25 @@ if (spec.drive) { process.stderr.write(`[runner] running drive() for ${spec.templateId}\n`); @@ -476,7 +478,7 @@ } else { process.stderr.write( `[runner] no drive() defined (CP-only scenario) -- nothing to do while it runs\n`, -@@ -190,15 +403,18 @@ +@@ -190,15 +405,18 @@ driveState = undefined as D; } @@ -498,7 +500,7 @@ // it a FAIL in a swept scenario is un-post-mortem-able: the sim container // is already stopped+removed by the time assert() reports, so this // capture is the only surviving record of what was (or wasn't) on the -@@ -218,6 +434,10 @@ +@@ -218,6 +436,10 @@ ); } @@ -509,7 +511,7 @@ const frames = parseLog(lines.join("\n")); const rec = new AssertRecorder(); -@@ -228,34 +448,53 @@ +@@ -228,34 +450,53 @@ frames, lines, rec, @@ -572,7 +574,7 @@ // --------------------------------------------------------------------------- // eslint-disable-next-line @typescript-eslint/no-explicit-any -@@ -265,8 +504,6 @@ +@@ -265,8 +506,6 @@ "remotetrigger-smartcharging": REMOTETRIGGER_SMARTCHARGING_SPECS, firmware: FIRMWARE_SPECS, authorize: AUTHORIZE_SPECS, @@ -581,7 +583,7 @@ all: [ ...CORE_SPECS, ...AUTHLIST_RESERVATION_SPECS, -@@ -286,66 +523,219 @@ +@@ -286,66 +525,241 @@ .map((spec) => [spec.templateId, spec]), ); @@ -688,13 +690,13 @@ + * + * A parallel-lane FAIL/ERROR that did not fail its isolated retry is a flake, + * not a failure; see {@link retryFailedOutcomesIsolated}. -+ */ -+function effectivelyFailed(outcome: ScenarioOutcome): boolean { -+ if (!isFailure(outcome.verdict)) return false; -+ return ( -+ outcome.isolatedRetry === undefined || -+ isFailure(outcome.isolatedRetry.verdict) -+ ); + */ ++function effectivelyFailed( ++ verdict: Verdict, ++ isolatedRetry?: RetryOutcome, ++): boolean { ++ if (!isFailure(verdict)) return false; ++ return isolatedRetry === undefined || isFailure(isolatedRetry.verdict); +} + +/** @@ -717,9 +719,13 @@ + | "unexpected-fail" + | "unexpected-pass"; + -+function standingOf(outcome: ScenarioOutcome): SweepStanding { -+ const failed = effectivelyFailed(outcome); -+ if (outcome.expected === undefined) { ++function standingOf( ++ verdict: Verdict, ++ expected: ExpectedFailureEntry | undefined, ++ isolatedRetry?: RetryOutcome, ++): SweepStanding { ++ const failed = effectivelyFailed(verdict, isolatedRetry); ++ if (expected === undefined) { + return failed ? "unexpected-fail" : "ok"; + } + if (failed) return "expected-fail"; @@ -733,6 +739,35 @@ + return "unexpected-pass"; +} + ++/** {@link standingOf} for a sweep outcome. The single-scenario path in `cli()` ++ * has no ScenarioOutcome and calls the fields form directly, which is the ++ * point of the split: the two paths cannot decide differently. */ ++function standingOfOutcome(outcome: ScenarioOutcome): SweepStanding { ++ return standingOf(outcome.verdict, outcome.expected, outcome.isolatedRetry); ++} ++ ++/** ++ * The driver's declaration, as the one sentence every report renders -- ++ * `undefined` when it declared nothing about this scenario. ++ * ++ * Written once because it was written three times: the markdown summary, the ++ * sweep's stderr list and the single-scenario line each rebuilt it, and the ++ * fourth copy had already drifted to a different separator. It is the same ++ * mistake unexpectedPassDetail() was extracted to fix, one level up. ++ * ++ * Narrowing `expected` here also removes the `?.` chains the call sites needed: ++ * they could not see that a standing of `expected-fail` implies an entry, so a ++ * broken invariant would have rendered the string "undefined" into a report. ++ */ ++function declarationNote(outcome: ScenarioOutcome): string | undefined { ++ const { expected } = outcome; ++ if (!expected) return undefined; ++ return effectivelyFailed(outcome.verdict, outcome.isolatedRetry) ++ ? `EXPECTED FAIL: ${expected.reason} (${expected.finding})` ++ : "UNEXPECTED PASS: declared expected-failing, but " + ++ unexpectedPassDetail(outcome.verdict, outcome.isolatedRetry); ++} ++ +/** + * Why an unexpected pass is one, in the words a reader needs to tell a fix + * from a flake from a contradiction from an unmeasured run -- without @@ -783,70 +818,66 @@ + * + * The scope table is consulted FIRST -- a NOT_APPLICABLE scenario never + * starts a container. - */ ++ */ async function runOneForSweep( spec: ScenarioSpec, cpId: string, ): Promise { -+ // Read before the run and carried on every branch, including the ones that -+ // never start a container: a declaration that only got attached to failures -+ // could never report the case the list exists to catch, which is the row -+ // that stopped failing. -+ const expected = await expectedFailureEntryFor(spec.templateId); ++ // What every branch below shares, stated once. `expected` is read BEFORE the ++ // run and carried even by the branches that never start a container: a ++ // declaration attached only to failures could never report the case the list ++ // exists to catch, which is the row that stopped failing. Hoisting it makes ++ // that structural instead of four sites each remembering. ++ const common = { ++ templateId: spec.templateId, ++ cpId, ++ checks: null, ++ failed: null, ++ skipped: null, ++ expected: await expectedFailureEntryFor(spec.templateId), ++ }; ++ + const scope = await scopeEntryFor(spec.templateId); + if (scope?.status === "NOT_APPLICABLE") { + process.stderr.write( + `[runner] === ${spec.templateId} NOT APPLICABLE (no container started): ${scope.reason}\n`, + ); -+ return { -+ templateId: spec.templateId, -+ cpId, -+ verdict: "NOT APPLICABLE", -+ checks: null, -+ failed: null, -+ skipped: null, -+ reason: scope.reason, -+ expected, -+ }; ++ return { ...common, verdict: "NOT APPLICABLE", reason: scope.reason }; + } + try { - const rec = await runScenario(spec, { cpId }); + const run = await runScenario(spec, { cpId }); + if (run.kind === "not-applicable") { -+ return { -+ templateId: spec.templateId, -+ cpId, -+ verdict: "NOT APPLICABLE", -+ checks: null, -+ failed: null, -+ skipped: null, -+ reason: run.reason, -+ expected, -+ }; ++ return { ...common, verdict: "NOT APPLICABLE", reason: run.reason }; + } return { - templateId: spec.templateId, - cpId, +- templateId: spec.templateId, +- cpId, - verdict: rec.verdict, - checks: rec.total, - failed: rec.failed, ++ ...common, + verdict: verdictForRecorder(run.rec), + checks: run.rec.total, + failed: run.rec.failed, + skipped: run.rec.skipped, -+ expected, }; } catch (err) { const message = -@@ -359,58 +749,47 @@ - verdict: "ERROR", - checks: null, - failed: null, -+ skipped: null, - errorMessage: message, -+ expected, - }; +@@ -353,64 +767,44 @@ + process.stderr.write( + `[runner] ERROR: ${spec.templateId} on ${cpId} threw before completing: ${message}\n`, + ); +- return { +- templateId: spec.templateId, +- cpId, +- verdict: "ERROR", +- checks: null, +- failed: null, +- errorMessage: message, +- }; ++ return { ...common, verdict: "ERROR", errorMessage: message }; } } @@ -920,7 +951,7 @@ ); for (const outcome of toRetry) { -@@ -432,13 +811,14 @@ +@@ -432,13 +826,14 @@ verdict: retryOutcome.verdict, checks: retryOutcome.checks, failed: retryOutcome.failed, @@ -937,7 +968,7 @@ : " -- CONFIRMED (fails isolated too, not a parallel-lane artifact)\n"), ); } -@@ -451,11 +831,43 @@ +@@ -451,11 +846,43 @@ return new Date().toISOString().replace(/\.\d{3}Z$/, "Z"); } @@ -986,25 +1017,20 @@ async function writeSummary( groupName: string, outcomes: ScenarioOutcome[], -@@ -465,45 +877,97 @@ +@@ -465,45 +892,92 @@ const rows = outcomes.map((o) => { const checks = o.checks === null ? "-" : String(o.checks); const failed = o.failed === null ? "-" : String(o.failed); - const base = `| ${o.templateId} | ${o.cpId} | ${o.verdict} | ${checks} | ${failed} |`; + const skipped = o.skipped === null ? "-" : String(o.skipped); -+ let verdict = o.reason ? `${o.verdict} (${o.reason})` : o.verdict; + // The declaration is rendered INTO the verdict cell rather than into a + // column of its own, so that a reader scanning for red sees, in the same + // glance, whether it was already known -- and, for an unexpected pass, -+ // sees the row that has to be deleted. -+ const standing = standingOf(o); -+ if (standing === "expected-fail") { -+ verdict = `${verdict} — EXPECTED FAIL: ${o.expected?.reason} (${o.expected?.finding})`; -+ } else if (standing === "unexpected-pass") { -+ verdict = -+ `${verdict} — UNEXPECTED PASS: declared expected-failing, but ` + -+ `${unexpectedPassDetail(o.verdict, o.isolatedRetry)}.`; -+ } ++ // sees the row that has to be looked at. ++ const note = declarationNote(o); ++ const verdict = ++ (o.reason ? `${o.verdict} (${o.reason})` : o.verdict) + ++ (note ? ` — ${note}.` : ""); + const base = `| ${o.templateId} | ${o.cpId} | ${verdict} | ${checks} | ${failed} | ${skipped} |`; if (!anyRetried) return base; if (!o.isolatedRetry) return `${base} - |`; @@ -1038,10 +1064,10 @@ + ); + } + const expectedFails = outcomes.filter( -+ (o) => standingOf(o) === "expected-fail", ++ (o) => standingOfOutcome(o) === "expected-fail", + ); + const unexpectedPasses = outcomes.filter( -+ (o) => standingOf(o) === "unexpected-pass", ++ (o) => standingOfOutcome(o) === "unexpected-pass", + ); + if (expectedFails.length > 0) { + notes.push( @@ -1099,7 +1125,7 @@ "", header, separator, -@@ -518,54 +982,53 @@ +@@ -518,54 +992,53 @@ } /** Sequential or parallel group sweep. Writes results/summary.md and exits @@ -1179,7 +1205,7 @@ ); } else { for (let i = 0; i < specs.length; i++) { -@@ -574,11 +1037,11 @@ +@@ -574,11 +1047,11 @@ } if (retryFailedIsolated) { @@ -1193,7 +1219,7 @@ "(a sequential sweep is already isolated).\n", ); } -@@ -588,45 +1051,85 @@ +@@ -588,45 +1061,80 @@ for (const o of outcomes) { const retrySuffix = o.isolatedRetry ? ` [isolated retry: ${o.isolatedRetry.verdict}${ @@ -1202,13 +1228,8 @@ }]` : ""; + const reasonSuffix = o.reason ? ` -- ${o.reason}` : ""; -+ const standing = standingOf(o); -+ const declaredSuffix = -+ standing === "expected-fail" -+ ? ` [EXPECTED FAIL: ${o.expected?.reason} (${o.expected?.finding})]` -+ : standing === "unexpected-pass" -+ ? ` [UNEXPECTED PASS: declared expected-failing, but ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}]` -+ : ""; ++ const note = declarationNote(o); ++ const declaredSuffix = note ? ` [${note}]` : ""; process.stderr.write( - ` ${o.verdict}: ${o.templateId} (${o.cpId}, ${o.checks ?? "-"} checks, ${o.failed ?? "-"} failed)${retrySuffix}\n`, + ` ${o.verdict}: ${o.templateId} (${o.cpId}, ${o.checks ?? "-"} checks, ${o.failed ?? "-"} failed, ${o.skipped ?? "-"} skipped)${retrySuffix}${declaredSuffix}${reasonSuffix}\n`, @@ -1227,20 +1248,20 @@ + // reads that through effectivelyFailed(), via standingOf(). const badOutcomes = outcomes.filter( - (o) => o.verdict !== "PASS" && o.isolatedRetry?.verdict !== "PASS", -+ (o) => standingOf(o) === "unexpected-fail", ++ (o) => standingOfOutcome(o) === "unexpected-fail", ); + const expectedFails = outcomes.filter( -+ (o) => standingOf(o) === "expected-fail", ++ (o) => standingOfOutcome(o) === "expected-fail", + ); + const unexpectedPasses = outcomes.filter( -+ (o) => standingOf(o) === "unexpected-pass", ++ (o) => standingOfOutcome(o) === "unexpected-pass", + ); ++ // A flake IS "failed, but not effectively" -- said through the shared ++ // predicate rather than spelled out again, which is what the comment above ++ // claims and what the previous hand-written copy quietly contradicted. const flakeCount = outcomes.filter( - (o) => o.verdict !== "PASS" && o.isolatedRetry?.verdict === "PASS", -+ (o) => -+ isFailure(o.verdict) && -+ o.isolatedRetry !== undefined && -+ !isFailure(o.isolatedRetry.verdict), ++ (o) => isFailure(o.verdict) && !effectivelyFailed(o.verdict, o.isolatedRetry), ).length; if (flakeCount > 0) { process.stderr.write( @@ -1294,7 +1315,7 @@ } // --------------------------------------------------------------------------- -@@ -642,6 +1145,8 @@ +@@ -642,6 +1150,8 @@ cpId: string; connector?: number; timeoutSecs?: number; @@ -1303,7 +1324,7 @@ } function requireValue(argv: string[], index: number, flag: string): string { -@@ -663,37 +1168,80 @@ +@@ -663,37 +1173,80 @@ return n; } @@ -1401,7 +1422,7 @@ if (argv[0] === "run-all") { group = "all"; -@@ -708,18 +1256,27 @@ +@@ -708,18 +1261,27 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1432,7 +1453,7 @@ process.exit(1); } -@@ -749,9 +1306,11 @@ +@@ -749,9 +1311,11 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1445,7 +1466,7 @@ process.exit(1); } } -@@ -764,35 +1323,495 @@ +@@ -764,35 +1328,495 @@ cpId, connector, timeoutSecs, @@ -1474,7 +1495,7 @@ + } + return [...seen].map(([templateId, group]) => ({ templateId, group })); +} -+ + +function listScenarios(argv: string[]): number { + let group: string | undefined; + let asJson = false; @@ -1657,7 +1678,7 @@ + ); + } + } - ++ + if (capabilities) { + const known = new Set(CSMS_OPERATION_ACTIONS); + const unknown = [...capabilities.operations].filter( @@ -1957,7 +1978,7 @@ } const spec = SPECS_BY_TEMPLATE_ID.get(args.templateId); -@@ -800,18 +1819,81 @@ +@@ -800,18 +1824,86 @@ process.stderr.write( `Unknown template id: ${args.templateId} (known: ${[...SPECS_BY_TEMPLATE_ID.keys()].join(", ")})\n`, ); @@ -1985,7 +2006,7 @@ ); - process.exit(1); - }); -+ return unexpectedPassExit(spec.templateId, "NOT APPLICABLE", expected); ++ return exitForSingleRun(spec.templateId, "NOT APPLICABLE", expected); + } + + const options: RunOptions = { @@ -1998,53 +2019,58 @@ + process.stderr.write( + `[runner] RESULT: ${spec.templateId} NOT APPLICABLE: ${run.reason}\n`, + ); -+ return unexpectedPassExit(spec.templateId, "NOT APPLICABLE", expected); ++ return exitForSingleRun(spec.templateId, "NOT APPLICABLE", expected); + } + -+ const verdict = verdictForRecorder(run.rec); -+ if (isFailure(verdict)) { -+ if (expected) { -+ process.stderr.write( -+ `[runner] ${spec.templateId} EXPECTED FAIL: ${expected.reason} ` + -+ `(${expected.finding}) -- declared by this driver, exit 0.\n`, -+ ); -+ return 0; -+ } -+ return 1; -+ } -+ // Only an ASSERTION failure is excusable here. A throw out of runScenario -- -+ // a bounded wait giving up, a driver blowing up -- propagates to -+ // bin/ocpp-tck.ts and exits 1 whatever the declaration says, where the sweep -+ // would have caught it as ERROR and excused it. The asymmetry is deliberate: -+ // this is the debugging mode, a stack trace is the thing worth having, and a -+ // declared finding about what a CSMS ANSWERS is not a licence to swallow a -+ // crash. CI runs the sweep, so no signal depends on the difference. -+ return unexpectedPassExit(spec.templateId, verdict, expected); ++ return exitForSingleRun( ++ spec.templateId, ++ verdictForRecorder(run.rec), ++ expected, ++ ); } + +/** -+ * Exit code for a single scenario that did NOT fail: 0 normally, 1 when the -+ * driver declared it expected-failing. ++ * Exit code for one scenario run by hand, through the SAME classifier the ++ * sweep's exit code reads. + * -+ * Shares unexpectedPassDetail() with the sweep rather than wording its own -+ * diagnosis. It did word its own, and the two disagreed on the case that -+ * matters most: a NOT APPLICABLE row that is also declared expected-failing is -+ * a contradiction between two tables, and this path told the reader to delete -+ * the entry while the sweep told them the contradiction was the thing to fix. ++ * It used to re-implement the rule in nested ifs, because standingOf() took a ++ * ScenarioOutcome this path does not have. That is the mistake ++ * unexpectedPassDetail() was reshaped to fix one level down -- the two paths ++ * had already disagreed once, about NOT APPLICABLE -- so the decision gets the ++ * same treatment as the message that explains it. + * -+ * A single run has no isolated retry, so there is no flake to distinguish -- -+ * one non-failure IS the whole evidence here, and the detail says which kind. ++ * A single run has no isolated retry, so no flake to weigh: one outcome IS the ++ * whole evidence, and unexpectedPassDetail() says which kind of evidence. ++ * ++ * Only an ASSERTION failure reaches this function. A throw out of runScenario ++ * -- a bounded wait giving up, a driver blowing up -- propagates to ++ * bin/ocpp-tck.ts and exits 1 whatever the declaration says, where the sweep ++ * would have caught it as ERROR and excused it. Deliberate: this is the ++ * debugging mode, a stack trace is the thing worth having, and a declared ++ * finding about what a CSMS ANSWERS is not a licence to swallow a crash. + */ -+function unexpectedPassExit( ++function exitForSingleRun( + templateId: string, + verdict: Verdict, + expected: ExpectedFailureEntry | undefined, +): number { -+ if (!expected) return 0; -+ process.stderr.write( -+ `[runner] ${templateId} UNEXPECTED PASS: this driver declares it ` + -+ `expected-failing (${expected.reason} -- ${expected.finding}), and ` + -+ `${unexpectedPassDetail(verdict)}.\n`, -+ ); -+ return 1; ++ switch (standingOf(verdict, expected)) { ++ case "ok": ++ return 0; ++ case "unexpected-fail": ++ return 1; ++ case "expected-fail": ++ process.stderr.write( ++ `[runner] ${templateId} EXPECTED FAIL: ${expected?.reason} ` + ++ `(${expected?.finding}) -- declared by this driver, exit 0.\n`, ++ ); ++ return 0; ++ case "unexpected-pass": ++ process.stderr.write( ++ `[runner] ${templateId} UNEXPECTED PASS: this driver declares it ` + ++ `expected-failing (${expected?.reason} -- ${expected?.finding}), ` + ++ `and ${unexpectedPassDetail(verdict)}.\n`, ++ ); ++ return 1; ++ } +} diff --git a/tck/driver.ts b/tck/driver.ts index b3cb9c3..38beef3 100644 --- a/tck/driver.ts +++ b/tck/driver.ts @@ -485,8 +485,17 @@ export type CsmsEnv = Readonly>; * * `T` must not itself be callable: resolution discriminates on `typeof`, so a * function-valued declaration would be indistinguishable from its own - * resolver. Both fields using it are objects, which is also what lets the two + * resolver. Every field using it is an object, which is also what lets the * resolvers below narrow the union with no cast. + * + * THAT LAST PROPERTY IS WHY THE RESOLVERS ARE NOT ONE GENERIC HELPER, which + * is otherwise the obvious de-duplication and has been proposed once. Factored + * out over an unconstrained `T`, the union becomes + * `((env) => T) | (T & Function)` and `typeof value === "function"` no longer + * narrows it -- tsc says "not all constituents are callable" and the helper + * needs a cast. Three short bodies that the compiler checks beat one shared + * body that it cannot, for a rule whose entire failure mode is a declaration + * being read as its own resolver. */ export type EnvDependent = T | ((env: CsmsEnv) => T); diff --git a/tck/main.ts b/tck/main.ts index b744ecc..acb7ecf 100644 --- a/tck/main.ts +++ b/tck/main.ts @@ -117,7 +117,9 @@ interface RunOptions { /** Verdict for one scenario. PASS/FAIL/ERROR keep their upstream meaning. */ type Verdict = "PASS" | "PARTIAL" | "FAIL" | "ERROR" | "NOT APPLICABLE"; -/** Only these two make the process exit non-zero. */ +/** The two verdicts that count as a failure. Whether one ends the process + * non-zero is {@link standingOf}'s answer, not this one's -- see the exit-code + * rule at the top of this file. */ function isFailure(verdict: Verdict): boolean { return verdict === "FAIL" || verdict === "ERROR"; } @@ -593,12 +595,12 @@ interface ScenarioOutcome extends RetryOutcome { * A parallel-lane FAIL/ERROR that did not fail its isolated retry is a flake, * not a failure; see {@link retryFailedOutcomesIsolated}. */ -function effectivelyFailed(outcome: ScenarioOutcome): boolean { - if (!isFailure(outcome.verdict)) return false; - return ( - outcome.isolatedRetry === undefined || - isFailure(outcome.isolatedRetry.verdict) - ); +function effectivelyFailed( + verdict: Verdict, + isolatedRetry?: RetryOutcome, +): boolean { + if (!isFailure(verdict)) return false; + return isolatedRetry === undefined || isFailure(isolatedRetry.verdict); } /** @@ -621,9 +623,13 @@ type SweepStanding = | "unexpected-fail" | "unexpected-pass"; -function standingOf(outcome: ScenarioOutcome): SweepStanding { - const failed = effectivelyFailed(outcome); - if (outcome.expected === undefined) { +function standingOf( + verdict: Verdict, + expected: ExpectedFailureEntry | undefined, + isolatedRetry?: RetryOutcome, +): SweepStanding { + const failed = effectivelyFailed(verdict, isolatedRetry); + if (expected === undefined) { return failed ? "unexpected-fail" : "ok"; } if (failed) return "expected-fail"; @@ -637,6 +643,35 @@ function standingOf(outcome: ScenarioOutcome): SweepStanding { return "unexpected-pass"; } +/** {@link standingOf} for a sweep outcome. The single-scenario path in `cli()` + * has no ScenarioOutcome and calls the fields form directly, which is the + * point of the split: the two paths cannot decide differently. */ +function standingOfOutcome(outcome: ScenarioOutcome): SweepStanding { + return standingOf(outcome.verdict, outcome.expected, outcome.isolatedRetry); +} + +/** + * The driver's declaration, as the one sentence every report renders -- + * `undefined` when it declared nothing about this scenario. + * + * Written once because it was written three times: the markdown summary, the + * sweep's stderr list and the single-scenario line each rebuilt it, and the + * fourth copy had already drifted to a different separator. It is the same + * mistake unexpectedPassDetail() was extracted to fix, one level up. + * + * Narrowing `expected` here also removes the `?.` chains the call sites needed: + * they could not see that a standing of `expected-fail` implies an entry, so a + * broken invariant would have rendered the string "undefined" into a report. + */ +function declarationNote(outcome: ScenarioOutcome): string | undefined { + const { expected } = outcome; + if (!expected) return undefined; + return effectivelyFailed(outcome.verdict, outcome.isolatedRetry) + ? `EXPECTED FAIL: ${expected.reason} (${expected.finding})` + : "UNEXPECTED PASS: declared expected-failing, but " + + unexpectedPassDetail(outcome.verdict, outcome.isolatedRetry); +} + /** * Why an unexpected pass is one, in the words a reader needs to tell a fix * from a flake from a contradiction from an unmeasured run -- without @@ -692,50 +727,39 @@ async function runOneForSweep( spec: ScenarioSpec, cpId: string, ): Promise { - // Read before the run and carried on every branch, including the ones that - // never start a container: a declaration that only got attached to failures - // could never report the case the list exists to catch, which is the row - // that stopped failing. - const expected = await expectedFailureEntryFor(spec.templateId); + // What every branch below shares, stated once. `expected` is read BEFORE the + // run and carried even by the branches that never start a container: a + // declaration attached only to failures could never report the case the list + // exists to catch, which is the row that stopped failing. Hoisting it makes + // that structural instead of four sites each remembering. + const common = { + templateId: spec.templateId, + cpId, + checks: null, + failed: null, + skipped: null, + expected: await expectedFailureEntryFor(spec.templateId), + }; + const scope = await scopeEntryFor(spec.templateId); if (scope?.status === "NOT_APPLICABLE") { process.stderr.write( `[runner] === ${spec.templateId} NOT APPLICABLE (no container started): ${scope.reason}\n`, ); - return { - templateId: spec.templateId, - cpId, - verdict: "NOT APPLICABLE", - checks: null, - failed: null, - skipped: null, - reason: scope.reason, - expected, - }; + return { ...common, verdict: "NOT APPLICABLE", reason: scope.reason }; } try { const run = await runScenario(spec, { cpId }); if (run.kind === "not-applicable") { - return { - templateId: spec.templateId, - cpId, - verdict: "NOT APPLICABLE", - checks: null, - failed: null, - skipped: null, - reason: run.reason, - expected, - }; + return { ...common, verdict: "NOT APPLICABLE", reason: run.reason }; } return { - templateId: spec.templateId, - cpId, + ...common, verdict: verdictForRecorder(run.rec), checks: run.rec.total, failed: run.rec.failed, skipped: run.rec.skipped, - expected, }; } catch (err) { const message = @@ -743,16 +767,7 @@ async function runOneForSweep( process.stderr.write( `[runner] ERROR: ${spec.templateId} on ${cpId} threw before completing: ${message}\n`, ); - return { - templateId: spec.templateId, - cpId, - verdict: "ERROR", - checks: null, - failed: null, - skipped: null, - errorMessage: message, - expected, - }; + return { ...common, verdict: "ERROR", errorMessage: message }; } } @@ -878,19 +893,14 @@ async function writeSummary( const checks = o.checks === null ? "-" : String(o.checks); const failed = o.failed === null ? "-" : String(o.failed); const skipped = o.skipped === null ? "-" : String(o.skipped); - let verdict = o.reason ? `${o.verdict} (${o.reason})` : o.verdict; // The declaration is rendered INTO the verdict cell rather than into a // column of its own, so that a reader scanning for red sees, in the same // glance, whether it was already known -- and, for an unexpected pass, - // sees the row that has to be deleted. - const standing = standingOf(o); - if (standing === "expected-fail") { - verdict = `${verdict} — EXPECTED FAIL: ${o.expected?.reason} (${o.expected?.finding})`; - } else if (standing === "unexpected-pass") { - verdict = - `${verdict} — UNEXPECTED PASS: declared expected-failing, but ` + - `${unexpectedPassDetail(o.verdict, o.isolatedRetry)}.`; - } + // sees the row that has to be looked at. + const note = declarationNote(o); + const verdict = + (o.reason ? `${o.verdict} (${o.reason})` : o.verdict) + + (note ? ` — ${note}.` : ""); const base = `| ${o.templateId} | ${o.cpId} | ${verdict} | ${checks} | ${failed} | ${skipped} |`; if (!anyRetried) return base; if (!o.isolatedRetry) return `${base} - |`; @@ -919,10 +929,10 @@ async function writeSummary( ); } const expectedFails = outcomes.filter( - (o) => standingOf(o) === "expected-fail", + (o) => standingOfOutcome(o) === "expected-fail", ); const unexpectedPasses = outcomes.filter( - (o) => standingOf(o) === "unexpected-pass", + (o) => standingOfOutcome(o) === "unexpected-pass", ); if (expectedFails.length > 0) { notes.push( @@ -1055,13 +1065,8 @@ async function runGroupSweep( }]` : ""; const reasonSuffix = o.reason ? ` -- ${o.reason}` : ""; - const standing = standingOf(o); - const declaredSuffix = - standing === "expected-fail" - ? ` [EXPECTED FAIL: ${o.expected?.reason} (${o.expected?.finding})]` - : standing === "unexpected-pass" - ? ` [UNEXPECTED PASS: declared expected-failing, but ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}]` - : ""; + const note = declarationNote(o); + const declaredSuffix = note ? ` [${note}]` : ""; process.stderr.write( ` ${o.verdict}: ${o.templateId} (${o.cpId}, ${o.checks ?? "-"} checks, ${o.failed ?? "-"} failed, ${o.skipped ?? "-"} skipped)${retrySuffix}${declaredSuffix}${reasonSuffix}\n`, ); @@ -1074,19 +1079,19 @@ async function runGroupSweep( // flake, not a real failure -- it does not fail the sweep. Everything below // reads that through effectivelyFailed(), via standingOf(). const badOutcomes = outcomes.filter( - (o) => standingOf(o) === "unexpected-fail", + (o) => standingOfOutcome(o) === "unexpected-fail", ); const expectedFails = outcomes.filter( - (o) => standingOf(o) === "expected-fail", + (o) => standingOfOutcome(o) === "expected-fail", ); const unexpectedPasses = outcomes.filter( - (o) => standingOf(o) === "unexpected-pass", + (o) => standingOfOutcome(o) === "unexpected-pass", ); + // A flake IS "failed, but not effectively" -- said through the shared + // predicate rather than spelled out again, which is what the comment above + // claims and what the previous hand-written copy quietly contradicted. const flakeCount = outcomes.filter( - (o) => - isFailure(o.verdict) && - o.isolatedRetry !== undefined && - !isFailure(o.isolatedRetry.verdict), + (o) => isFailure(o.verdict) && !effectivelyFailed(o.verdict, o.isolatedRetry), ).length; if (flakeCount > 0) { process.stderr.write( @@ -1834,7 +1839,7 @@ export async function cli(argv: string[]): Promise { process.stderr.write( `[runner] RESULT: ${spec.templateId} NOT APPLICABLE (no container started): ${scope.reason}\n`, ); - return unexpectedPassExit(spec.templateId, "NOT APPLICABLE", expected); + return exitForSingleRun(spec.templateId, "NOT APPLICABLE", expected); } const options: RunOptions = { @@ -1847,53 +1852,58 @@ export async function cli(argv: string[]): Promise { process.stderr.write( `[runner] RESULT: ${spec.templateId} NOT APPLICABLE: ${run.reason}\n`, ); - return unexpectedPassExit(spec.templateId, "NOT APPLICABLE", expected); + return exitForSingleRun(spec.templateId, "NOT APPLICABLE", expected); } - const verdict = verdictForRecorder(run.rec); - if (isFailure(verdict)) { - if (expected) { - process.stderr.write( - `[runner] ${spec.templateId} EXPECTED FAIL: ${expected.reason} ` + - `(${expected.finding}) -- declared by this driver, exit 0.\n`, - ); - return 0; - } - return 1; - } - // Only an ASSERTION failure is excusable here. A throw out of runScenario -- - // a bounded wait giving up, a driver blowing up -- propagates to - // bin/ocpp-tck.ts and exits 1 whatever the declaration says, where the sweep - // would have caught it as ERROR and excused it. The asymmetry is deliberate: - // this is the debugging mode, a stack trace is the thing worth having, and a - // declared finding about what a CSMS ANSWERS is not a licence to swallow a - // crash. CI runs the sweep, so no signal depends on the difference. - return unexpectedPassExit(spec.templateId, verdict, expected); + return exitForSingleRun( + spec.templateId, + verdictForRecorder(run.rec), + expected, + ); } /** - * Exit code for a single scenario that did NOT fail: 0 normally, 1 when the - * driver declared it expected-failing. + * Exit code for one scenario run by hand, through the SAME classifier the + * sweep's exit code reads. + * + * It used to re-implement the rule in nested ifs, because standingOf() took a + * ScenarioOutcome this path does not have. That is the mistake + * unexpectedPassDetail() was reshaped to fix one level down -- the two paths + * had already disagreed once, about NOT APPLICABLE -- so the decision gets the + * same treatment as the message that explains it. * - * Shares unexpectedPassDetail() with the sweep rather than wording its own - * diagnosis. It did word its own, and the two disagreed on the case that - * matters most: a NOT APPLICABLE row that is also declared expected-failing is - * a contradiction between two tables, and this path told the reader to delete - * the entry while the sweep told them the contradiction was the thing to fix. + * A single run has no isolated retry, so no flake to weigh: one outcome IS the + * whole evidence, and unexpectedPassDetail() says which kind of evidence. * - * A single run has no isolated retry, so there is no flake to distinguish -- - * one non-failure IS the whole evidence here, and the detail says which kind. + * Only an ASSERTION failure reaches this function. A throw out of runScenario + * -- a bounded wait giving up, a driver blowing up -- propagates to + * bin/ocpp-tck.ts and exits 1 whatever the declaration says, where the sweep + * would have caught it as ERROR and excused it. Deliberate: this is the + * debugging mode, a stack trace is the thing worth having, and a declared + * finding about what a CSMS ANSWERS is not a licence to swallow a crash. */ -function unexpectedPassExit( +function exitForSingleRun( templateId: string, verdict: Verdict, expected: ExpectedFailureEntry | undefined, ): number { - if (!expected) return 0; - process.stderr.write( - `[runner] ${templateId} UNEXPECTED PASS: this driver declares it ` + - `expected-failing (${expected.reason} -- ${expected.finding}), and ` + - `${unexpectedPassDetail(verdict)}.\n`, - ); - return 1; + switch (standingOf(verdict, expected)) { + case "ok": + return 0; + case "unexpected-fail": + return 1; + case "expected-fail": + process.stderr.write( + `[runner] ${templateId} EXPECTED FAIL: ${expected?.reason} ` + + `(${expected?.finding}) -- declared by this driver, exit 0.\n`, + ); + return 0; + case "unexpected-pass": + process.stderr.write( + `[runner] ${templateId} UNEXPECTED PASS: this driver declares it ` + + `expected-failing (${expected?.reason} -- ${expected?.finding}), ` + + `and ${unexpectedPassDetail(verdict)}.\n`, + ); + return 1; + } } From 46785e280b262f3e48c463939a45c7f3212510e3 Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 14 Aug 2026 11:16:03 +0200 Subject: [PATCH 07/11] fix(runner)!: an expected failure excuses an answer, never a crash A declared scenario that ERRORed was excused exactly like one that FAILed, so a container that refused to boot, a bounded wait giving up or a driver throwing all passed as the documented finding. That is the continue-on-error blindness this mechanism replaced, narrowed to one scenario but just as blind to the KIND of failure -- and it defeats the point, because the one row CitrineOS's lane is allowed to be red on is now the one row that could hide anything. An entry says what a CSMS ANSWERS: its reason cites a handler, a status mapping, a field that comes back wrong. An ERROR is the scenario never getting an answer to be wrong about. So `expected-fail` now needs the decisive verdict to be exactly FAIL; a declared scenario that errors is an ordinary unexpected failure, reported on its own line because the reaction is the opposite one -- the entry is almost certainly still good and the crash is the new thing. Decisive means the isolated retry when there is one, which is the existing doctrine rather than a new rule: parallel lanes are not isolated and the retry exists to re-run without contention. So ERROR-then-FAIL-isolated is the declared failure (the ERROR was the lane), and FAIL-then-ERROR-isolated is not. THE RULE MOVES TO tck/standing.ts, and that is what makes the change defensible. Inside the runner it was reachable only by starting a container and driving a live CSMS, so the rows that decide whether a red build is honest were untestable in practice. As a pure function over verdict x retry x declaration it is a table, and tests/expected-failure-standing.ts asserts all 16 rows offline -- wired into `bun run verify`, `bun run test` and CI. Each of the four ways to break the rule was mutation-tested to fail on its own row and no other. BREAKING CHANGE: a driver declaring `expectedFailures` no longer has an ERROR excused on those scenarios. Nothing changes for a driver that declares none, and nothing changes for a declared scenario that fails the way it says it does. --- .github/workflows/ci.yml | 7 + AGENTS.md | 12 +- CONTRIBUTING.md | 7 + README.md | 5 + VENDOR.md | 3 +- package.json | 2 +- patches/tck/main.ts.patch | 257 ++++++++++++----------------- tck/expected.ts | 8 + tck/main.ts | 199 +++++++++------------- tck/standing.ts | 199 ++++++++++++++++++++++ tests/expected-failure-standing.ts | 140 ++++++++++++++++ tools/verify.sh | 5 + types/tck/driver.d.ts | 11 +- types/tck/expected.d.ts | 8 + types/tck/standing.d.ts | 103 ++++++++++++ 15 files changed, 683 insertions(+), 283 deletions(-) create mode 100644 tck/standing.ts create mode 100644 tests/expected-failure-standing.ts create mode 100644 types/tck/standing.d.ts diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 02e46ba..26de842 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -65,6 +65,13 @@ jobs: - name: Driver scope follows the environment run: bun tests/driver-env-scope.ts + # In-process for the same reason: the rule that decides whether a red + # sweep ends the build is not reachable from a shell without running a + # scenario, and the rows that matter would each need a CSMS engineered to + # fail a chosen way. + - name: Exit-code rule holds + run: bun tests/expected-failure-standing.ts + - name: Core names no CSMS run: bash tests/generic-core.sh diff --git a/AGENTS.md b/AGENTS.md index 8c69101..084ac17 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,6 +34,7 @@ bun run check:driver:steve bun run check:driver:citrineos bun run check:driver:citrineos-v1 # the same driver's other release line bun tests/driver-env-scope.ts +bun tests/expected-failure-standing.ts ``` then `bun run verify` once before committing. @@ -79,9 +80,14 @@ There is no unit-test framework and no `*.test.ts`. `tests/` holds offline guards, each with a header stating the property it protects. `bun run test` chains them — note `bun test` is Bun's own runner and finds nothing here. -Shell is the default; `tests/driver-env-scope.ts` is TypeScript because the -property it asserts (a driver's declarations follow the env they are *resolved* -with) is unreachable through the CLI, which can only ever pass `process.env`. +Shell is the default, and the two TypeScript ones are TypeScript because what +they assert is unreachable through the CLI. `driver-env-scope.ts`: a driver's +declarations follow the env they are *resolved* with, where the CLI can only +ever pass `process.env`. `expected-failure-standing.ts`: the rule that decides +whether a red sweep ends the build, which from a shell would cost a container +per row — and, for the rows that matter, a CSMS engineered to fail a chosen +scenario a chosen way. `tck/standing.ts` is a module of its own so that guard +can be a table. ## Two boundaries the guards enforce diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8381a24..829f861 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -198,6 +198,13 @@ that half is not an inconvenience — it is what the mechanism is for. Without i the list only ever grows, and an entry outlives the defect it documents. When one fires, delete the entry or re-word it to say what is still true. +**An entry excuses an answer, never a crash.** A declared scenario that `ERROR`s +— container never started, bounded wait gave up, driver threw — still fails the +sweep, reported as `DECLARED, BUT ERRORED`. It never got an answer out of the +CSMS, so it cannot be the finding your `reason` describes, and a job that went +green on it would be blind to exactly the kind of breakage it exists to catch. +Your entry is probably still good; the crash is the new thing. + Three rules, and they are the difference between a reviewed list and a mute: - `reason` names the mechanism. "Known red" is the observation being explained, diff --git a/README.md b/README.md index 2e0cada..b60e2e8 100644 --- a/README.md +++ b/README.md @@ -203,6 +203,11 @@ deleted the day upstream fixes the defect, instead of outliving it. There is deliberately no "expected flaky" — a scenario that sometimes passes has a timing bug, and the bug is the thing to fix. +**A declared scenario that `ERROR`s also fails the sweep**, as +`DECLARED, BUT ERRORED`. An entry excuses what a CSMS *answers*; an `ERROR` is +the scenario never getting an answer, and a green build there would be blind to +the kind of breakage the mechanism exists to catch. + ## Environment | Variable | Default | Meaning | diff --git a/VENDOR.md b/VENDOR.md index 4cdade1..62a93c4 100644 --- a/VENDOR.md +++ b/VENDOR.md @@ -57,13 +57,14 @@ strongest available demonstration that the core names no CSMS. | `tck/specs/index.ts` | `upstream-verbatim` | `scripts/steve-verify/runner/specs/index.ts` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `—` | | `tck/assert.ts` | `upstream-patched` | `src/cp/application/verification/assert.ts` | `2431f5f6c0df997d4d821d9af55689c1f0f2df199de1e9e4ed6f3fbaad4fc89e` | `76d0f293db4f6ecc5768affea0ca76d2841c147fa3687dbea1f9e950bfbd9298` | `patches/tck/assert.ts.patch` | | `tck/sim.ts` | `upstream-patched` | `scripts/steve-verify/runner/sim.ts` | `2bf2f78afe3434e7139cd62c3ff6d70f02defd39dd700611e7c5f7614260cd35` | `45f897f273ea73227bb9b30766127eb9ecae0a319ab9048cbc30428fafff7f32` | `patches/tck/sim.ts.patch` | -| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `7a0e8b6a19a13a8ce80a426ba63b77b8d23715d78f2ddd8dc4e7a6943fea5f15` | `patches/tck/main.ts.patch` | +| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `5febf8e25fd435d930798d2dd1d1771c448efb479d84fe9693a9aa27281f0aff` | `patches/tck/main.ts.patch` | | `tck/driver.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/index.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/driver-registry.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/capabilities.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/scope.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/expected.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | +| `tck/standing.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/time.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/unverifiable.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/wait.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | diff --git a/package.json b/package.json index 03f9c01..830912c 100644 --- a/package.json +++ b/package.json @@ -121,7 +121,7 @@ "e2e": "bun bin/ocpp-tck.ts run-all --parallel --retry-failed-isolated; all=$?; bun bin/ocpp-tck.ts run-all --group authorize; auth=$?; exit $(( all || auth ))", "e2e:smoke": "bun bin/ocpp-tck.ts run cert16-tc023-1-authorize-invalid && bun bin/ocpp-tck.ts run cert16-tc023-2-authorize-expired && bun bin/ocpp-tck.ts run cert16-tc023-3-authorize-blocked && bun bin/ocpp-tck.ts run cert16-tc066-get-composite-schedule && bun bin/ocpp-tck.ts run cert16-tc003-charging-plugin-first", "check:fixture-tags": "bash tools/extract-fixture-tags.sh --diff", - "test": "bash tests/vendor-integrity.sh && bash tests/generic-core.sh && bash tests/spec-invariants.sh && bun tests/driver-env-scope.ts" + "test": "bash tests/vendor-integrity.sh && bash tests/generic-core.sh && bash tests/spec-invariants.sh && bun tests/driver-env-scope.ts && bun tests/expected-failure-standing.ts" }, "devDependencies": { "@types/bun": "1.3.14", diff --git a/patches/tck/main.ts.patch b/patches/tck/main.ts.patch index 7c93359..93ee1ba 100644 --- a/patches/tck/main.ts.patch +++ b/patches/tck/main.ts.patch @@ -1,6 +1,6 @@ --- /tmp/ocpp-upstream/scripts/steve-verify/runner/main.ts 2026-07-31 20:05:48 +++ tck/main.ts 2026-07-31 20:12:10 -@@ -1,39 +1,81 @@ +@@ -1,39 +1,95 @@ #!/usr/bin/env bun +// Derived from shiv3/ocpp-cp-simulator scripts/steve-verify/runner/main.ts @ 604054adb0d7d7129a26a5f1ad2d5fdc290d1ca1 (Apache-2.0). Modified: the STEVE_DRIVER=api|ui selection is replaced by a CSMS driver loaded through ./driver-registry; a per-driver scope table (./scope) is consulted BEFORE any container starts and yields the NOT APPLICABLE verdict; UnsupportedOperationError (./driver) thrown out of drive() degrades to NOT APPLICABLE with a stderr WARNING; the PARTIAL verdict and the `skipped` summary column were added; the exit code is non-zero only for FAIL/ERROR; parallel lanes derive from the resolved station list instead of the fixed CERTCP1..3 trio; the SteVe capability probe is dropped. + @@ -45,11 +45,16 @@ + * driver, and the two stopped being the same thing when expected.ts arrived. + * A FAIL the driver declared expected exits 0 -- it is a finding already + * written down, not news -- and a PASS it declared expected exits 1, because -+ * a list that cannot shrink is a mute. So: ++ * a list that cannot shrink is a mute. An ERROR is never excused: a ++ * declaration covers what a CSMS ANSWERS, and an ERROR is the scenario never ++ * getting an answer. So: + * -+ * exit non-zero <= an undeclared FAIL/ERROR, or a declared one that passed. ++ * exit non-zero <= an undeclared FAIL/ERROR, or a declared scenario that ++ * passed, or a declared scenario that errored. + * -+ * A driver that declares nothing keeps the original rule exactly. ++ * The rule itself lives in ./standing, where it is a pure function and a guard ++ * can assert the whole table without a container. A driver that declares ++ * nothing keeps the original rule exactly. */ import { mkdirSync } from "node:fs"; @@ -79,6 +84,15 @@ + type ScopeTable, +} from "./scope"; +import { ++ declaredButErroredDetail, ++ effectivelyFailed, ++ isFailure, ++ standingOf, ++ unexpectedPassDetail, ++ type SweepStanding, ++ type Verdict, ++} from "./standing"; ++import { + unsupportedChargingProfiles, + unsupportedReservations, +} from "./capabilities"; @@ -108,7 +122,7 @@ import { AUTHLIST_RESERVATION_SPECS, AUTHORIZE_SPECS, -@@ -43,13 +85,25 @@ +@@ -43,13 +99,25 @@ } from "./specs/index"; import type { ScenarioSpec } from "./spec-types"; import { sleep } from "./util"; @@ -139,20 +153,10 @@ // --------------------------------------------------------------------------- // Runner core // --------------------------------------------------------------------------- -@@ -60,95 +114,244 @@ +@@ -60,95 +128,234 @@ timeoutSecs?: number; } -+/** Verdict for one scenario. PASS/FAIL/ERROR keep their upstream meaning. */ -+type Verdict = "PASS" | "PARTIAL" | "FAIL" | "ERROR" | "NOT APPLICABLE"; -+ -+/** The two verdicts that count as a failure. Whether one ends the process -+ * non-zero is {@link standingOf}'s answer, not this one's -- see the exit-code -+ * rule at the top of this file. */ -+function isFailure(verdict: Verdict): boolean { -+ return verdict === "FAIL" || verdict === "ERROR"; -+} -+ +/** 0 FAIL + >=1 SKIPPED is PARTIAL; anything with a FAIL is FAIL. */ +function verdictForRecorder(rec: AssertRecorder): Verdict { + if (rec.failed > 0) return "FAIL"; @@ -445,7 +449,7 @@ 30_000, ); } catch (err) { -@@ -176,13 +379,25 @@ +@@ -176,13 +383,25 @@ if (spec.drive) { process.stderr.write(`[runner] running drive() for ${spec.templateId}\n`); @@ -478,7 +482,7 @@ } else { process.stderr.write( `[runner] no drive() defined (CP-only scenario) -- nothing to do while it runs\n`, -@@ -190,15 +405,18 @@ +@@ -190,15 +409,18 @@ driveState = undefined as D; } @@ -500,7 +504,7 @@ // it a FAIL in a swept scenario is un-post-mortem-able: the sim container // is already stopped+removed by the time assert() reports, so this // capture is the only surviving record of what was (or wasn't) on the -@@ -218,6 +436,10 @@ +@@ -218,6 +440,10 @@ ); } @@ -511,7 +515,7 @@ const frames = parseLog(lines.join("\n")); const rec = new AssertRecorder(); -@@ -228,34 +450,53 @@ +@@ -228,34 +454,53 @@ frames, lines, rec, @@ -574,7 +578,7 @@ // --------------------------------------------------------------------------- // eslint-disable-next-line @typescript-eslint/no-explicit-any -@@ -265,8 +506,6 @@ +@@ -265,8 +510,6 @@ "remotetrigger-smartcharging": REMOTETRIGGER_SMARTCHARGING_SPECS, firmware: FIRMWARE_SPECS, authorize: AUTHORIZE_SPECS, @@ -583,7 +587,7 @@ all: [ ...CORE_SPECS, ...AUTHLIST_RESERVATION_SPECS, -@@ -286,66 +525,241 @@ +@@ -286,66 +529,154 @@ .map((spec) => [spec.templateId, spec]), ); @@ -674,6 +678,13 @@ + expected?: ExpectedFailureEntry; } ++/** {@link standingOf} for a sweep outcome. The single-scenario path in `cli()` ++ * has no ScenarioOutcome and calls the fields form directly, which is the ++ * point of the split: the two paths cannot decide differently. */ ++function standingOfOutcome(outcome: ScenarioOutcome): SweepStanding { ++ return standingOf(outcome.verdict, outcome.expected, outcome.isolatedRetry?.verdict); ++} ++ /** - * Runs one scenario for a group sweep, isolating a thrown exception (e.g. - * waitForCondition/waitActiveTxPk's fail-hard timeout rejection propagating @@ -681,72 +692,6 @@ - * sweep -- bash's per-scenario isolation comes for free from each - * run-scenario.sh invocation being a separate subprocess; this is the - * same-process equivalent. -+ * Did this scenario really fail -- after the isolated retry has had its say? -+ * -+ * ONE definition, deliberately, because there are now two consumers: the exit -+ * code, and the comparison against the driver's expected-failure list. Written -+ * twice they would drift, and the drift would be silent in the worst possible -+ * place -- a row excused by one rule and counted by the other. -+ * -+ * A parallel-lane FAIL/ERROR that did not fail its isolated retry is a flake, -+ * not a failure; see {@link retryFailedOutcomesIsolated}. - */ -+function effectivelyFailed( -+ verdict: Verdict, -+ isolatedRetry?: RetryOutcome, -+): boolean { -+ if (!isFailure(verdict)) return false; -+ return isolatedRetry === undefined || isFailure(isolatedRetry.verdict); -+} -+ -+/** -+ * How an outcome lands against what the driver declared. This -- not the -+ * verdict -- is what the exit code reads. -+ * -+ * `expected-fail` is the whole point of the list: a finding already written -+ * down is not news, and muting it one scenario at a time is what lets the -+ * other 46 be reported at all. -+ * -+ * `unexpected-pass` is what keeps the list from becoming the job-level mute it -+ * replaced. It fires on a declared scenario that did NOT effectively fail, -+ * which includes the case where it failed in parallel and passed its isolated -+ * retry -- there is no "expected flaky", so an entry that passes any way at -+ * all is an entry to delete or to re-word. -+ */ -+type SweepStanding = -+ | "ok" -+ | "expected-fail" -+ | "unexpected-fail" -+ | "unexpected-pass"; -+ -+function standingOf( -+ verdict: Verdict, -+ expected: ExpectedFailureEntry | undefined, -+ isolatedRetry?: RetryOutcome, -+): SweepStanding { -+ const failed = effectivelyFailed(verdict, isolatedRetry); -+ if (expected === undefined) { -+ return failed ? "unexpected-fail" : "ok"; -+ } -+ if (failed) return "expected-fail"; -+ // Everything else is an unexpected pass, INCLUDING NOT APPLICABLE. That one -+ // is not merely theoretical: check-driver rejects an id declared both ways, -+ // but only a driver that has been checked, and the runtime -+ // UnsupportedOperationError escape produces the same contradiction after the -+ // fact, when no offline check can have seen it. It is surfaced rather than -+ // tolerated because "it never ran" is not "it was fixed" -- -+ // unexpectedPassDetail() is where the difference gets said. -+ return "unexpected-pass"; -+} -+ -+/** {@link standingOf} for a sweep outcome. The single-scenario path in `cli()` -+ * has no ScenarioOutcome and calls the fields form directly, which is the -+ * point of the split: the two paths cannot decide differently. */ -+function standingOfOutcome(outcome: ScenarioOutcome): SweepStanding { -+ return standingOf(outcome.verdict, outcome.expected, outcome.isolatedRetry); -+} -+ -+/** + * The driver's declaration, as the one sentence every report renders -- + * `undefined` when it declared nothing about this scenario. + * @@ -758,57 +703,29 @@ + * Narrowing `expected` here also removes the `?.` chains the call sites needed: + * they could not see that a standing of `expected-fail` implies an entry, so a + * broken invariant would have rendered the string "undefined" into a report. -+ */ + */ +function declarationNote(outcome: ScenarioOutcome): string | undefined { -+ const { expected } = outcome; ++ const { expected, verdict } = outcome; ++ const retryVerdict = outcome.isolatedRetry?.verdict; + if (!expected) return undefined; -+ return effectivelyFailed(outcome.verdict, outcome.isolatedRetry) -+ ? `EXPECTED FAIL: ${expected.reason} (${expected.finding})` -+ : "UNEXPECTED PASS: declared expected-failing, but " + -+ unexpectedPassDetail(outcome.verdict, outcome.isolatedRetry); -+} -+ -+/** -+ * Why an unexpected pass is one, in the words a reader needs to tell a fix -+ * from a flake from a contradiction from an unmeasured run -- without -+ * re-reading this file. -+ * -+ * The wording carries real weight, because the action it implies is DELETING A -+ * RECORDED FINDING. Only one of the four cases below is actually evidence that -+ * the CSMS was fixed; saying so for the other three would talk a maintainer -+ * into throwing away a finding that still holds. -+ * -+ * Takes the fields rather than the outcome, so that the single-scenario path -+ * -- which has no ScenarioOutcome and no isolated retry -- diagnoses the same -+ * situation the same way. It used to have a message of its own, and the two -+ * disagreed about the NOT APPLICABLE case. -+ */ -+function unexpectedPassDetail( -+ verdict: Verdict, -+ isolatedRetry?: RetryOutcome, -+): string { -+ if (verdict === "NOT APPLICABLE") { -+ return ( -+ "it never ran (NOT APPLICABLE) -- the scope table and the " + -+ "expected-failure list contradict each other, so the contradiction is " + -+ "what to fix, and deleting the entry may be the wrong half" -+ ); -+ } -+ if (verdict === "PARTIAL") { -+ return ( -+ "it came back PARTIAL -- at least one check was SKIPPED because the " + -+ "driver could not evaluate it, so this is NOT evidence that the finding " + -+ "is fixed; find out which check degraded before deleting the entry" -+ ); -+ } -+ if (isolatedRetry !== undefined && isFailure(verdict)) { -+ return ( -+ `it failed in the parallel lane and ${isolatedRetry.verdict} on its ` + -+ "isolated retry -- either upstream fixed it, or the scenario is flaky " + -+ "and the flake is the bug to fix" -+ ); ++ switch (standingOf(verdict, expected, retryVerdict)) { ++ case "expected-fail": ++ return `EXPECTED FAIL: ${expected.reason} (${expected.finding})`; ++ case "unexpected-pass": ++ return ( ++ "UNEXPECTED PASS: declared expected-failing, but " + ++ unexpectedPassDetail(verdict, retryVerdict) ++ ); ++ case "unexpected-fail": ++ return ( ++ `DECLARED, BUT ERRORED (${expected.reason} -- ${expected.finding}): ` + ++ declaredButErroredDetail(verdict, retryVerdict) ++ ); ++ case "ok": ++ // Unreachable: `expected` is non-undefined here, and standingOf() only ++ // answers "ok" for an undeclared scenario. ++ return undefined; + } -+ return `it came back ${verdict} outright -- the finding looks fixed`; +} + +/** @@ -865,7 +782,7 @@ }; } catch (err) { const message = -@@ -353,64 +767,44 @@ +@@ -353,64 +684,44 @@ process.stderr.write( `[runner] ERROR: ${spec.templateId} on ${cpId} threw before completing: ${message}\n`, ); @@ -951,7 +868,7 @@ ); for (const outcome of toRetry) { -@@ -432,13 +826,14 @@ +@@ -432,13 +743,14 @@ verdict: retryOutcome.verdict, checks: retryOutcome.checks, failed: retryOutcome.failed, @@ -968,7 +885,7 @@ : " -- CONFIRMED (fails isolated too, not a parallel-lane artifact)\n"), ); } -@@ -451,11 +846,43 @@ +@@ -451,11 +763,43 @@ return new Date().toISOString().replace(/\.\d{3}Z$/, "Z"); } @@ -1017,7 +934,7 @@ async function writeSummary( groupName: string, outcomes: ScenarioOutcome[], -@@ -465,45 +892,92 @@ +@@ -465,45 +809,110 @@ const rows = outcomes.map((o) => { const checks = o.checks === null ? "-" : String(o.checks); const failed = o.failed === null ? "-" : String(o.failed); @@ -1081,6 +998,24 @@ + ), + ); + } ++ const declaredButErrored = outcomes.filter( ++ (o) => o.expected !== undefined && standingOfOutcome(o) === "unexpected-fail", ++ ); ++ if (declaredButErrored.length > 0) { ++ notes.push( ++ "", ++ `${declaredButErrored.length} DECLARED, BUT ERRORED — declared ` + ++ "expected-failing and did not produce the declared failure; it never " + ++ "got an answer out of the CSMS at all. **This fails the sweep**: an " + ++ "entry excuses what a CSMS answers, never a crash, and a job that went " + ++ "green on this would be blind to the kind of breakage it exists to " + ++ "catch. The entries below are probably still good:", ++ ...declaredButErrored.map( ++ (o) => ++ `- \`${o.templateId}\` — ${declaredButErroredDetail(o.verdict, o.isolatedRetry?.verdict)}`, ++ ), ++ ); ++ } + if (unexpectedPasses.length > 0) { + notes.push( + "", @@ -1089,7 +1024,7 @@ + "known-red entry gets deleted when it stops being true, instead of " + + "outliving the defect it documents.", + ...unexpectedPasses.map( -+ (o) => `- \`${o.templateId}\` — ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}`, ++ (o) => `- \`${o.templateId}\` — ${unexpectedPassDetail(o.verdict, o.isolatedRetry?.verdict)}`, + ), + ); + } @@ -1125,7 +1060,7 @@ "", header, separator, -@@ -518,54 +992,53 @@ +@@ -518,54 +927,53 @@ } /** Sequential or parallel group sweep. Writes results/summary.md and exits @@ -1205,7 +1140,7 @@ ); } else { for (let i = 0; i < specs.length; i++) { -@@ -574,11 +1047,11 @@ +@@ -574,11 +982,11 @@ } if (retryFailedIsolated) { @@ -1219,7 +1154,7 @@ "(a sequential sweep is already isolated).\n", ); } -@@ -588,45 +1061,80 @@ +@@ -588,45 +996,96 @@ for (const o of outcomes) { const retrySuffix = o.isolatedRetry ? ` [isolated retry: ${o.isolatedRetry.verdict}${ @@ -1256,12 +1191,18 @@ + const unexpectedPasses = outcomes.filter( + (o) => standingOfOutcome(o) === "unexpected-pass", + ); ++ // A subset of badOutcomes, reported on its own line because it reads as a ++ // regression of the DECLARATION when it is nothing of the sort: the entry is ++ // almost certainly still good and something new stopped the scenario ever ++ // reaching the CSMS. Without this the maintainer sees a red build on the one ++ // row they were told would be red, and goes looking in the wrong place. ++ const declaredButErrored = badOutcomes.filter((o) => o.expected !== undefined); + // A flake IS "failed, but not effectively" -- said through the shared + // predicate rather than spelled out again, which is what the comment above + // claims and what the previous hand-written copy quietly contradicted. const flakeCount = outcomes.filter( - (o) => o.verdict !== "PASS" && o.isolatedRetry?.verdict === "PASS", -+ (o) => isFailure(o.verdict) && !effectivelyFailed(o.verdict, o.isolatedRetry), ++ (o) => isFailure(o.verdict) && !effectivelyFailed(o.verdict, o.isolatedRetry?.verdict), ).length; if (flakeCount > 0) { process.stderr.write( @@ -1274,16 +1215,26 @@ + `[runner] ${expectedFails.length} scenario(s) in group '${groupName}' failed as this driver declared they would -- not a build failure; see ${summaryPath}\n`, + ); + } ++ if (declaredButErrored.length > 0) { ++ process.stderr.write( ++ `[runner] ${declaredButErrored.length} scenario(s) in group '${groupName}' are declared expected-failing and ERRORED, which is not the declared failure:\n`, ++ ); ++ for (const o of declaredButErrored) { ++ process.stderr.write( ++ ` ${o.templateId}: ${declaredButErroredDetail(o.verdict, o.isolatedRetry?.verdict)}.\n`, ++ ); ++ } ++ } + // Reported BEFORE the ordinary failures and counted separately, because the + // two need opposite fixes: an unexpected failure is a defect to chase, an -+ // unexpected pass is a line to delete. ++ // unexpected pass is a line to look at. + if (unexpectedPasses.length > 0) { + process.stderr.write( + `[runner] ${unexpectedPasses.length} scenario(s) in group '${groupName}' are declared expected-failing and did NOT fail:\n`, + ); + for (const o of unexpectedPasses) { + process.stderr.write( -+ ` ${o.templateId}: ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}.\n`, ++ ` ${o.templateId}: ${unexpectedPassDetail(o.verdict, o.isolatedRetry?.verdict)}.\n`, + ); + } + } @@ -1315,7 +1266,7 @@ } // --------------------------------------------------------------------------- -@@ -642,6 +1150,8 @@ +@@ -642,6 +1101,8 @@ cpId: string; connector?: number; timeoutSecs?: number; @@ -1324,7 +1275,7 @@ } function requireValue(argv: string[], index: number, flag: string): string { -@@ -663,37 +1173,80 @@ +@@ -663,37 +1124,80 @@ return n; } @@ -1422,7 +1373,7 @@ if (argv[0] === "run-all") { group = "all"; -@@ -708,18 +1261,27 @@ +@@ -708,18 +1212,27 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1453,7 +1404,7 @@ process.exit(1); } -@@ -749,9 +1311,11 @@ +@@ -749,9 +1262,11 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1466,7 +1417,7 @@ process.exit(1); } } -@@ -764,35 +1328,495 @@ +@@ -764,35 +1279,495 @@ cpId, connector, timeoutSecs, @@ -1978,7 +1929,7 @@ } const spec = SPECS_BY_TEMPLATE_ID.get(args.templateId); -@@ -800,18 +1824,86 @@ +@@ -800,18 +1775,86 @@ process.stderr.write( `Unknown template id: ${args.templateId} (known: ${[...SPECS_BY_TEMPLATE_ID.keys()].join(", ")})\n`, ); diff --git a/tck/expected.ts b/tck/expected.ts index 5f5c604..378dffc 100644 --- a/tck/expected.ts +++ b/tck/expected.ts @@ -25,6 +25,14 @@ * UNEXPECTED PASS, with a non-zero exit -- which is how a row gets deleted * when upstream fixes the defect, instead of outliving it. * + * WHAT AN ENTRY DOES NOT EXCUSE: a crash. It says what a CSMS *answers* -- + * its `reason` cites a handler, a status mapping, a field that comes back + * wrong -- and a scenario that ERRORs never got an answer to be wrong about. + * So a declared scenario that errors still fails the sweep, and is reported + * separately from an ordinary failure because the entry is probably still good + * and the crash is the new thing. The rule is {@link ./standing}'s + * `standingOf`, and `tests/expected-failure-standing.ts` asserts the table. + * * RULES FOR A DRIVER AUTHOR EDITING ITS LIST * - `reason` names the mechanism, cited, the way a scope row's does. "Fails * on this CSMS" is not a reason; it is the observation being explained. diff --git a/tck/main.ts b/tck/main.ts index acb7ecf..a420556 100644 --- a/tck/main.ts +++ b/tck/main.ts @@ -26,11 +26,16 @@ * driver, and the two stopped being the same thing when expected.ts arrived. * A FAIL the driver declared expected exits 0 -- it is a finding already * written down, not news -- and a PASS it declared expected exits 1, because - * a list that cannot shrink is a mute. So: + * a list that cannot shrink is a mute. An ERROR is never excused: a + * declaration covers what a CSMS ANSWERS, and an ERROR is the scenario never + * getting an answer. So: * - * exit non-zero <= an undeclared FAIL/ERROR, or a declared one that passed. + * exit non-zero <= an undeclared FAIL/ERROR, or a declared scenario that + * passed, or a declared scenario that errored. * - * A driver that declares nothing keeps the original rule exactly. + * The rule itself lives in ./standing, where it is a pure function and a guard + * can assert the whole table without a container. A driver that declares + * nothing keeps the original rule exactly. */ import { mkdirSync } from "node:fs"; @@ -57,6 +62,15 @@ import { type ScopeStatus, type ScopeTable, } from "./scope"; +import { + declaredButErroredDetail, + effectivelyFailed, + isFailure, + standingOf, + unexpectedPassDetail, + type SweepStanding, + type Verdict, +} from "./standing"; import { unsupportedChargingProfiles, unsupportedReservations, @@ -114,16 +128,6 @@ interface RunOptions { timeoutSecs?: number; } -/** Verdict for one scenario. PASS/FAIL/ERROR keep their upstream meaning. */ -type Verdict = "PASS" | "PARTIAL" | "FAIL" | "ERROR" | "NOT APPLICABLE"; - -/** The two verdicts that count as a failure. Whether one ends the process - * non-zero is {@link standingOf}'s answer, not this one's -- see the exit-code - * rule at the top of this file. */ -function isFailure(verdict: Verdict): boolean { - return verdict === "FAIL" || verdict === "ERROR"; -} - /** 0 FAIL + >=1 SKIPPED is PARTIAL; anything with a FAIL is FAIL. */ function verdictForRecorder(rec: AssertRecorder): Verdict { if (rec.failed > 0) return "FAIL"; @@ -584,70 +588,11 @@ interface ScenarioOutcome extends RetryOutcome { expected?: ExpectedFailureEntry; } -/** - * Did this scenario really fail -- after the isolated retry has had its say? - * - * ONE definition, deliberately, because there are now two consumers: the exit - * code, and the comparison against the driver's expected-failure list. Written - * twice they would drift, and the drift would be silent in the worst possible - * place -- a row excused by one rule and counted by the other. - * - * A parallel-lane FAIL/ERROR that did not fail its isolated retry is a flake, - * not a failure; see {@link retryFailedOutcomesIsolated}. - */ -function effectivelyFailed( - verdict: Verdict, - isolatedRetry?: RetryOutcome, -): boolean { - if (!isFailure(verdict)) return false; - return isolatedRetry === undefined || isFailure(isolatedRetry.verdict); -} - -/** - * How an outcome lands against what the driver declared. This -- not the - * verdict -- is what the exit code reads. - * - * `expected-fail` is the whole point of the list: a finding already written - * down is not news, and muting it one scenario at a time is what lets the - * other 46 be reported at all. - * - * `unexpected-pass` is what keeps the list from becoming the job-level mute it - * replaced. It fires on a declared scenario that did NOT effectively fail, - * which includes the case where it failed in parallel and passed its isolated - * retry -- there is no "expected flaky", so an entry that passes any way at - * all is an entry to delete or to re-word. - */ -type SweepStanding = - | "ok" - | "expected-fail" - | "unexpected-fail" - | "unexpected-pass"; - -function standingOf( - verdict: Verdict, - expected: ExpectedFailureEntry | undefined, - isolatedRetry?: RetryOutcome, -): SweepStanding { - const failed = effectivelyFailed(verdict, isolatedRetry); - if (expected === undefined) { - return failed ? "unexpected-fail" : "ok"; - } - if (failed) return "expected-fail"; - // Everything else is an unexpected pass, INCLUDING NOT APPLICABLE. That one - // is not merely theoretical: check-driver rejects an id declared both ways, - // but only a driver that has been checked, and the runtime - // UnsupportedOperationError escape produces the same contradiction after the - // fact, when no offline check can have seen it. It is surfaced rather than - // tolerated because "it never ran" is not "it was fixed" -- - // unexpectedPassDetail() is where the difference gets said. - return "unexpected-pass"; -} - /** {@link standingOf} for a sweep outcome. The single-scenario path in `cli()` * has no ScenarioOutcome and calls the fields form directly, which is the * point of the split: the two paths cannot decide differently. */ function standingOfOutcome(outcome: ScenarioOutcome): SweepStanding { - return standingOf(outcome.verdict, outcome.expected, outcome.isolatedRetry); + return standingOf(outcome.verdict, outcome.expected, outcome.isolatedRetry?.verdict); } /** @@ -664,55 +609,27 @@ function standingOfOutcome(outcome: ScenarioOutcome): SweepStanding { * broken invariant would have rendered the string "undefined" into a report. */ function declarationNote(outcome: ScenarioOutcome): string | undefined { - const { expected } = outcome; + const { expected, verdict } = outcome; + const retryVerdict = outcome.isolatedRetry?.verdict; if (!expected) return undefined; - return effectivelyFailed(outcome.verdict, outcome.isolatedRetry) - ? `EXPECTED FAIL: ${expected.reason} (${expected.finding})` - : "UNEXPECTED PASS: declared expected-failing, but " + - unexpectedPassDetail(outcome.verdict, outcome.isolatedRetry); -} - -/** - * Why an unexpected pass is one, in the words a reader needs to tell a fix - * from a flake from a contradiction from an unmeasured run -- without - * re-reading this file. - * - * The wording carries real weight, because the action it implies is DELETING A - * RECORDED FINDING. Only one of the four cases below is actually evidence that - * the CSMS was fixed; saying so for the other three would talk a maintainer - * into throwing away a finding that still holds. - * - * Takes the fields rather than the outcome, so that the single-scenario path - * -- which has no ScenarioOutcome and no isolated retry -- diagnoses the same - * situation the same way. It used to have a message of its own, and the two - * disagreed about the NOT APPLICABLE case. - */ -function unexpectedPassDetail( - verdict: Verdict, - isolatedRetry?: RetryOutcome, -): string { - if (verdict === "NOT APPLICABLE") { - return ( - "it never ran (NOT APPLICABLE) -- the scope table and the " + - "expected-failure list contradict each other, so the contradiction is " + - "what to fix, and deleting the entry may be the wrong half" - ); - } - if (verdict === "PARTIAL") { - return ( - "it came back PARTIAL -- at least one check was SKIPPED because the " + - "driver could not evaluate it, so this is NOT evidence that the finding " + - "is fixed; find out which check degraded before deleting the entry" - ); - } - if (isolatedRetry !== undefined && isFailure(verdict)) { - return ( - `it failed in the parallel lane and ${isolatedRetry.verdict} on its ` + - "isolated retry -- either upstream fixed it, or the scenario is flaky " + - "and the flake is the bug to fix" - ); + switch (standingOf(verdict, expected, retryVerdict)) { + case "expected-fail": + return `EXPECTED FAIL: ${expected.reason} (${expected.finding})`; + case "unexpected-pass": + return ( + "UNEXPECTED PASS: declared expected-failing, but " + + unexpectedPassDetail(verdict, retryVerdict) + ); + case "unexpected-fail": + return ( + `DECLARED, BUT ERRORED (${expected.reason} -- ${expected.finding}): ` + + declaredButErroredDetail(verdict, retryVerdict) + ); + case "ok": + // Unreachable: `expected` is non-undefined here, and standingOf() only + // answers "ok" for an undeclared scenario. + return undefined; } - return `it came back ${verdict} outright -- the finding looks fixed`; } /** @@ -946,6 +863,24 @@ async function writeSummary( ), ); } + const declaredButErrored = outcomes.filter( + (o) => o.expected !== undefined && standingOfOutcome(o) === "unexpected-fail", + ); + if (declaredButErrored.length > 0) { + notes.push( + "", + `${declaredButErrored.length} DECLARED, BUT ERRORED — declared ` + + "expected-failing and did not produce the declared failure; it never " + + "got an answer out of the CSMS at all. **This fails the sweep**: an " + + "entry excuses what a CSMS answers, never a crash, and a job that went " + + "green on this would be blind to the kind of breakage it exists to " + + "catch. The entries below are probably still good:", + ...declaredButErrored.map( + (o) => + `- \`${o.templateId}\` — ${declaredButErroredDetail(o.verdict, o.isolatedRetry?.verdict)}`, + ), + ); + } if (unexpectedPasses.length > 0) { notes.push( "", @@ -954,7 +889,7 @@ async function writeSummary( "known-red entry gets deleted when it stops being true, instead of " + "outliving the defect it documents.", ...unexpectedPasses.map( - (o) => `- \`${o.templateId}\` — ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}`, + (o) => `- \`${o.templateId}\` — ${unexpectedPassDetail(o.verdict, o.isolatedRetry?.verdict)}`, ), ); } @@ -1087,11 +1022,17 @@ async function runGroupSweep( const unexpectedPasses = outcomes.filter( (o) => standingOfOutcome(o) === "unexpected-pass", ); + // A subset of badOutcomes, reported on its own line because it reads as a + // regression of the DECLARATION when it is nothing of the sort: the entry is + // almost certainly still good and something new stopped the scenario ever + // reaching the CSMS. Without this the maintainer sees a red build on the one + // row they were told would be red, and goes looking in the wrong place. + const declaredButErrored = badOutcomes.filter((o) => o.expected !== undefined); // A flake IS "failed, but not effectively" -- said through the shared // predicate rather than spelled out again, which is what the comment above // claims and what the previous hand-written copy quietly contradicted. const flakeCount = outcomes.filter( - (o) => isFailure(o.verdict) && !effectivelyFailed(o.verdict, o.isolatedRetry), + (o) => isFailure(o.verdict) && !effectivelyFailed(o.verdict, o.isolatedRetry?.verdict), ).length; if (flakeCount > 0) { process.stderr.write( @@ -1103,16 +1044,26 @@ async function runGroupSweep( `[runner] ${expectedFails.length} scenario(s) in group '${groupName}' failed as this driver declared they would -- not a build failure; see ${summaryPath}\n`, ); } + if (declaredButErrored.length > 0) { + process.stderr.write( + `[runner] ${declaredButErrored.length} scenario(s) in group '${groupName}' are declared expected-failing and ERRORED, which is not the declared failure:\n`, + ); + for (const o of declaredButErrored) { + process.stderr.write( + ` ${o.templateId}: ${declaredButErroredDetail(o.verdict, o.isolatedRetry?.verdict)}.\n`, + ); + } + } // Reported BEFORE the ordinary failures and counted separately, because the // two need opposite fixes: an unexpected failure is a defect to chase, an - // unexpected pass is a line to delete. + // unexpected pass is a line to look at. if (unexpectedPasses.length > 0) { process.stderr.write( `[runner] ${unexpectedPasses.length} scenario(s) in group '${groupName}' are declared expected-failing and did NOT fail:\n`, ); for (const o of unexpectedPasses) { process.stderr.write( - ` ${o.templateId}: ${unexpectedPassDetail(o.verdict, o.isolatedRetry)}.\n`, + ` ${o.templateId}: ${unexpectedPassDetail(o.verdict, o.isolatedRetry?.verdict)}.\n`, ); } } diff --git a/tck/standing.ts b/tck/standing.ts new file mode 100644 index 0000000..6c8e71d --- /dev/null +++ b/tck/standing.ts @@ -0,0 +1,199 @@ +// Copyright 2026 Julien Herr +// SPDX-License-Identifier: Apache-2.0 +/** + * standing.ts -- the exit-code rule, as a pure function of one scenario's + * verdicts and what the driver declared about it. + * + * WHY IT HAS ITS OWN MODULE. It is the whole load-bearing decision of the + * expected-failure mechanism -- the thing that says a red CI job means + * something -- and inside the runner it was reachable only by starting a + * container and driving a live CSMS. Here it is a total function over five + * verdicts and one optional retry, so `tests/expected-failure-standing.ts` + * can assert the entire truth table offline, including the rows a sweep would + * take a docker image and ten minutes to reach once each. + * + * It names no CSMS and imports nothing but the declaration type, which is what + * keeps it on the core side of the boundary tests/generic-core.sh enforces. + */ +import type { ExpectedFailureEntry } from "./expected"; + +/** Verdict for one scenario. PASS/FAIL/ERROR keep their upstream meaning. + * + * The distinction that matters below is FAIL vs ERROR: FAIL is an assertion + * that ran and disagreed with the CSMS, ERROR is the scenario never getting + * that far. */ +export type Verdict = "PASS" | "PARTIAL" | "FAIL" | "ERROR" | "NOT APPLICABLE"; + +/** The two verdicts that count as a failure. Whether one ends the process + * non-zero is {@link standingOf}'s answer, not this one's. */ +export function isFailure(verdict: Verdict): boolean { + return verdict === "FAIL" || verdict === "ERROR"; +} + +/** + * The verdict to believe when a scenario ran twice. + * + * The isolated retry is the arbiter, and that is the existing doctrine rather + * than a new rule: parallel lanes are not isolated from each other, the runner + * says so on every parallel sweep, and the retry exists precisely to re-run a + * failure with nothing else contending. So a scenario that ERRORed under + * contention and FAILed cleanly on its own is a FAIL -- the ERROR was the + * lane, not the CSMS. + */ +function decisiveVerdict(verdict: Verdict, retryVerdict?: Verdict): Verdict { + return retryVerdict ?? verdict; +} + +/** + * Did this scenario really fail -- after the isolated retry has had its say? + * + * ONE definition, deliberately, because there are several consumers: the exit + * code, the flake count, and the comparison against the driver's + * expected-failure list. Written twice they would drift, and the drift would + * be silent in the worst possible place -- a row excused by one rule and + * counted by the other. + * + * A parallel-lane FAIL/ERROR that did not fail its isolated retry is a flake, + * not a failure. + */ +export function effectivelyFailed( + verdict: Verdict, + retryVerdict?: Verdict, +): boolean { + if (!isFailure(verdict)) return false; + return retryVerdict === undefined || isFailure(retryVerdict); +} + +/** + * How an outcome lands against what the driver declared. This -- not the + * verdict -- is what the exit code reads. + * + * `expected-fail` is the point of the list: a finding already written down is + * not news, and excusing it one scenario at a time is what lets the other 46 + * be reported at all. + * + * `unexpected-pass` is what keeps the list from becoming the job-level mute it + * replaced. It fires on a declared scenario that did NOT effectively fail, + * which includes the case where it failed in parallel and passed its isolated + * retry -- there is no "expected flaky", so an entry that passes any way at + * all is an entry to look at. + * + * `unexpected-fail` covers BOTH an undeclared failure and a declared scenario + * that ERRORed. See {@link standingOf} for why the second belongs here. + */ +export type SweepStanding = + | "ok" + | "expected-fail" + | "unexpected-fail" + | "unexpected-pass"; + +/** + * A DECLARATION EXCUSES AN ANSWER, NEVER A CRASH. + * + * An expected-failure entry says what a CSMS *replies* -- its `reason` cites a + * handler, a status mapping, a field that comes back wrong. A scenario that + * ERRORs never got a reply to be wrong about: the container did not start, a + * bounded wait gave up, the driver threw. Treating that as the declared + * failure is how a job goes green on a breakage of a completely different + * nature from the one it documents -- which is the `continue-on-error` + * blindness this whole mechanism replaced, narrowed to one scenario but just + * as blind to the KIND of failure. + * + * So `expected-fail` needs the decisive verdict to be exactly FAIL. A declared + * scenario that errors is an ordinary unexpected failure, and the report says + * that its declaration is probably still good and the crash is the new thing. + * + * The full table, with `d` = decisive verdict ({@link decisiveVerdict}): + * + * | declared | effectively failed | d | standing | + * |----------|--------------------|--------|-----------------| + * | no | no | any | ok | + * | no | yes | any | unexpected-fail | + * | yes | yes | FAIL | expected-fail | + * | yes | yes | ERROR | unexpected-fail | + * | yes | no | any | unexpected-pass | + */ +export function standingOf( + verdict: Verdict, + expected: ExpectedFailureEntry | undefined, + retryVerdict?: Verdict, +): SweepStanding { + const failed = effectivelyFailed(verdict, retryVerdict); + if (expected === undefined) { + return failed ? "unexpected-fail" : "ok"; + } + if (failed) { + return decisiveVerdict(verdict, retryVerdict) === "FAIL" + ? "expected-fail" + : "unexpected-fail"; + } + // Everything else is an unexpected pass, INCLUDING NOT APPLICABLE. That one + // is not merely theoretical: check-driver rejects an id declared both ways, + // but only for a driver that has been checked, and the runtime + // UnsupportedOperationError escape produces the same contradiction after the + // fact, when no offline check can have seen it. It is surfaced rather than + // tolerated because "it never ran" is not "it was fixed" -- + // unexpectedPassDetail() is where the difference gets said. + return "unexpected-pass"; +} + +/** + * Why an unexpected pass is one, in the words a reader needs to tell a fix + * from a flake from a contradiction from an unmeasured run -- without reading + * this file. + * + * The wording carries real weight, because the action it implies is DELETING A + * RECORDED FINDING. Only one of the cases below is actually evidence that the + * CSMS was fixed; saying so for the others would talk a maintainer into + * throwing away a finding that still holds. + */ +export function unexpectedPassDetail( + verdict: Verdict, + retryVerdict?: Verdict, +): string { + if (verdict === "NOT APPLICABLE") { + return ( + "it never ran (NOT APPLICABLE) -- the scope table and the " + + "expected-failure list contradict each other, so the contradiction is " + + "what to fix, and deleting the entry may be the wrong half" + ); + } + if (verdict === "PARTIAL") { + return ( + "it came back PARTIAL -- at least one check was SKIPPED because the " + + "driver could not evaluate it, so this is NOT evidence that the finding " + + "is fixed; find out which check degraded before deleting the entry" + ); + } + if (retryVerdict !== undefined && isFailure(verdict)) { + return ( + `it failed in the parallel lane and ${retryVerdict} on its isolated ` + + "retry -- either upstream fixed it, or the scenario is flaky and the " + + "flake is the bug to fix" + ); + } + return `it came back ${verdict} outright -- the finding looks fixed`; +} + +/** + * Why a DECLARED scenario is nevertheless an unexpected failure: it errored. + * + * Separated from the ordinary failure report because the two need opposite + * reactions. An undeclared failure is a finding to investigate and perhaps + * declare; this one almost certainly leaves the declaration intact and points + * at something new that stopped the scenario reaching the CSMS at all. + */ +export function declaredButErroredDetail( + verdict: Verdict, + retryVerdict?: Verdict, +): string { + const where = + retryVerdict === undefined + ? `it came back ${verdict}` + : `it came back ${verdict} and ${retryVerdict} on its isolated retry`; + return ( + `${where} -- an expected-failure entry excuses what a CSMS ANSWERS, and ` + + "an ERROR is the scenario never getting an answer. The declaration is " + + "probably still good; the crash is the new thing" + ); +} diff --git a/tests/expected-failure-standing.ts b/tests/expected-failure-standing.ts new file mode 100644 index 0000000..da99e60 --- /dev/null +++ b/tests/expected-failure-standing.ts @@ -0,0 +1,140 @@ +// Copyright 2026 Julien Herr +// SPDX-License-Identifier: Apache-2.0 +/** + * expected-failure-standing.ts -- the exit-code rule, whole, offline. + * + * THE PROPERTY: a scenario's standing is a total function of three things -- + * the verdict, the isolated retry's verdict if there was one, and whether the + * driver declared the scenario expected-failing. Every row of that table is + * asserted here, including the ones that decide whether a CI job is honest. + * + * WHY IT IS A GUARD AND NOT A SWEEP. Reaching these rows through the runner + * means a docker image, a live CSMS and about ten minutes, and reaching them + * ON PURPOSE means engineering a CSMS that fails a chosen scenario a chosen + * way -- which nobody will do for a rule that is one `||` away from being + * wrong in a direction that turns a red build green. `tck/standing.ts` exists + * as a separate module so this file can be a table. + * + * WHY IT IS TYPESCRIPT, like tests/driver-env-scope.ts: the rule is not + * reachable through the CLI without running a scenario, which is the whole + * point above. + * + * THE ROW THAT MATTERS MOST is `declared + ERROR`. An expected-failure entry + * says what a CSMS ANSWERS; an ERROR is the scenario never getting an answer, + * so excusing it would let a container that refuses to boot pass as the + * documented finding -- exactly the blindness the whole mechanism replaced. + */ +import type { ExpectedFailureEntry } from "../tck/expected"; +import { + effectivelyFailed, + standingOf, + unexpectedPassDetail, + type SweepStanding, + type Verdict, +} from "../tck/standing"; + +const failures: string[] = []; + +const DECLARED: ExpectedFailureEntry = { + reason: "the CSMS answers X where the spec requires Y", + finding: "acme/acme#1", +}; + +interface Row { + verdict: Verdict; + retry?: Verdict; + declared: boolean; + expect: SweepStanding; + /** Why this row is what it is -- printed when it breaks, so a failure names + * the rule rather than the coordinates. */ + because: string; +} + +const TABLE: Row[] = [ + // --- nothing declared: the original rule, unchanged -------------------- + { verdict: "PASS", declared: false, expect: "ok", because: "a pass is a pass" }, + { verdict: "PARTIAL", declared: false, expect: "ok", because: "a check that could not be evaluated is not a defect" }, + { verdict: "NOT APPLICABLE", declared: false, expect: "ok", because: "out of scope for this CSMS" }, + { verdict: "FAIL", declared: false, expect: "unexpected-fail", because: "an undeclared failure is the thing a TCK exists to report" }, + { verdict: "ERROR", declared: false, expect: "unexpected-fail", because: "an undeclared error is equally a failure" }, + + // --- declared, and it failed as declared ------------------------------- + { verdict: "FAIL", declared: true, expect: "expected-fail", because: "the declared finding reproduced" }, + { verdict: "FAIL", retry: "FAIL", declared: true, expect: "expected-fail", because: "it reproduced isolated too, which is the strongest form" }, + + // --- declared, and it ERRORED: a declaration excuses an answer, not a + // crash. These four rows are the finding this guard was written for. + { verdict: "ERROR", declared: true, expect: "unexpected-fail", because: "an ERROR never produced the answer the entry describes" }, + { verdict: "ERROR", retry: "ERROR", declared: true, expect: "unexpected-fail", because: "errored isolated too -- still not the declared failure" }, + { verdict: "FAIL", retry: "ERROR", declared: true, expect: "unexpected-fail", because: "the isolated retry is the arbiter, and it crashed" }, + { verdict: "ERROR", retry: "FAIL", declared: true, expect: "expected-fail", because: "the isolated retry is the arbiter: the parallel ERROR was the lane, and isolated it reproduced the declared failure" }, + + // --- declared, and it did NOT fail: the half that lets the list shrink -- + { verdict: "PASS", declared: true, expect: "unexpected-pass", because: "the finding looks fixed, so the entry must be deleted or re-worded" }, + { verdict: "PARTIAL", declared: true, expect: "unexpected-pass", because: "a declared row that stopped being measurable is not a declared row that still holds" }, + { verdict: "NOT APPLICABLE", declared: true, expect: "unexpected-pass", because: "the scope table and the list contradict each other" }, + { verdict: "FAIL", retry: "PASS", declared: true, expect: "unexpected-pass", because: "there is no 'expected flaky' -- passing any way at all is an entry to look at" }, + { verdict: "ERROR", retry: "PASS", declared: true, expect: "unexpected-pass", because: "same, from the ERROR side" }, +]; + +for (const row of TABLE) { + const got = standingOf( + row.verdict, + row.declared ? DECLARED : undefined, + row.retry, + ); + const label = + `${row.declared ? "declared" : "undeclared"} ${row.verdict}` + + (row.retry ? ` -> retry ${row.retry}` : ""); + if (got !== row.expect) { + failures.push( + `${label}: expected "${row.expect}", got "${got}" -- ${row.because}.`, + ); + } +} + +// The two rules the table above is built on, asserted directly so a break says +// WHICH one moved rather than listing every row that depended on it. +check( + effectivelyFailed("FAIL") && !effectivelyFailed("FAIL", "PASS"), + "a parallel failure that passes its isolated retry is no longer a flake.", +); +check( + !effectivelyFailed("PASS") && !effectivelyFailed("PARTIAL"), + "a non-failure verdict is now treated as a failure.", +); + +// The message an UNEXPECTED PASS carries decides whether a maintainer deletes +// a still-valid finding, so the two cases that are NOT evidence of a fix must +// not claim to be. +check( + !unexpectedPassDetail("PARTIAL").includes("looks fixed"), + "a PARTIAL now reads as evidence the finding is fixed; it is evidence of " + + "nothing, because the check that caught it may be the one that SKIPPED.", +); +check( + !unexpectedPassDetail("NOT APPLICABLE").includes("looks fixed"), + "a NOT APPLICABLE now reads as evidence the finding is fixed; it means the " + + "scenario never ran.", +); +check( + unexpectedPassDetail("PASS").includes("looks fixed"), + "an outright PASS no longer reads as evidence the finding is fixed, which " + + "is the one case where it is.", +); + +function check(condition: boolean, failure: string): void { + if (!condition) failures.push(failure); +} + +if (failures.length > 0) { + process.stderr.write("FAIL: the exit-code rule does not hold.\n"); + for (const failure of failures) process.stderr.write(` - ${failure}\n`); + process.exit(1); +} + +process.stdout.write( + `Exit-code rule holds across all ${TABLE.length} verdict/retry/declaration ` + + "combinations (a declaration excuses an answer, never a crash, and never " + + "a scenario that stopped failing).\n", +); diff --git a/tools/verify.sh b/tools/verify.sh index 04b7564..b4aaeb7 100755 --- a/tools/verify.sh +++ b/tools/verify.sh @@ -47,6 +47,11 @@ run "driver scope: citrineos" bun run check:driver:citrineos # driver claims unverified -- and the v1 table is the derived one. run "driver scope: citrineos (v1)" bun run check:driver:citrineos-v1 run "driver scope follows the env" bun tests/driver-env-scope.ts +# The exit-code rule, whole. Pure and offline because tck/standing.ts is a +# module of its own -- reaching these rows through a sweep would take a +# container per row, and engineering a CSMS that fails a chosen scenario a +# chosen way for the rows that matter. +run "exit-code rule holds" bun tests/expected-failure-standing.ts run "core is CSMS-neutral" bash tests/generic-core.sh run "vendored files match VENDOR.md" bash tests/vendor-integrity.sh run "scenario invariants" bash tests/spec-invariants.sh diff --git a/types/tck/driver.d.ts b/types/tck/driver.d.ts index 42ee3b1..c456758 100644 --- a/types/tck/driver.d.ts +++ b/types/tck/driver.d.ts @@ -367,8 +367,17 @@ export type CsmsEnv = Readonly>; * * `T` must not itself be callable: resolution discriminates on `typeof`, so a * function-valued declaration would be indistinguishable from its own - * resolver. Both fields using it are objects, which is also what lets the two + * resolver. Every field using it is an object, which is also what lets the * resolvers below narrow the union with no cast. + * + * THAT LAST PROPERTY IS WHY THE RESOLVERS ARE NOT ONE GENERIC HELPER, which + * is otherwise the obvious de-duplication and has been proposed once. Factored + * out over an unconstrained `T`, the union becomes + * `((env) => T) | (T & Function)` and `typeof value === "function"` no longer + * narrows it -- tsc says "not all constituents are callable" and the helper + * needs a cast. Three short bodies that the compiler checks beat one shared + * body that it cannot, for a rule whose entire failure mode is a declaration + * being read as its own resolver. */ export type EnvDependent = T | ((env: CsmsEnv) => T); /** diff --git a/types/tck/expected.d.ts b/types/tck/expected.d.ts index 191d511..9489ca2 100644 --- a/types/tck/expected.d.ts +++ b/types/tck/expected.d.ts @@ -23,6 +23,14 @@ * UNEXPECTED PASS, with a non-zero exit -- which is how a row gets deleted * when upstream fixes the defect, instead of outliving it. * + * WHAT AN ENTRY DOES NOT EXCUSE: a crash. It says what a CSMS *answers* -- + * its `reason` cites a handler, a status mapping, a field that comes back + * wrong -- and a scenario that ERRORs never got an answer to be wrong about. + * So a declared scenario that errors still fails the sweep, and is reported + * separately from an ordinary failure because the entry is probably still good + * and the crash is the new thing. The rule is {@link ./standing}'s + * `standingOf`, and `tests/expected-failure-standing.ts` asserts the table. + * * RULES FOR A DRIVER AUTHOR EDITING ITS LIST * - `reason` names the mechanism, cited, the way a scope row's does. "Fails * on this CSMS" is not a reason; it is the observation being explained. diff --git a/types/tck/standing.d.ts b/types/tck/standing.d.ts new file mode 100644 index 0000000..7855873 --- /dev/null +++ b/types/tck/standing.d.ts @@ -0,0 +1,103 @@ +/** + * standing.ts -- the exit-code rule, as a pure function of one scenario's + * verdicts and what the driver declared about it. + * + * WHY IT HAS ITS OWN MODULE. It is the whole load-bearing decision of the + * expected-failure mechanism -- the thing that says a red CI job means + * something -- and inside the runner it was reachable only by starting a + * container and driving a live CSMS. Here it is a total function over five + * verdicts and one optional retry, so `tests/expected-failure-standing.ts` + * can assert the entire truth table offline, including the rows a sweep would + * take a docker image and ten minutes to reach once each. + * + * It names no CSMS and imports nothing but the declaration type, which is what + * keeps it on the core side of the boundary tests/generic-core.sh enforces. + */ +import type { ExpectedFailureEntry } from "./expected"; +/** Verdict for one scenario. PASS/FAIL/ERROR keep their upstream meaning. + * + * The distinction that matters below is FAIL vs ERROR: FAIL is an assertion + * that ran and disagreed with the CSMS, ERROR is the scenario never getting + * that far. */ +export type Verdict = "PASS" | "PARTIAL" | "FAIL" | "ERROR" | "NOT APPLICABLE"; +/** The two verdicts that count as a failure. Whether one ends the process + * non-zero is {@link standingOf}'s answer, not this one's. */ +export declare function isFailure(verdict: Verdict): boolean; +/** + * Did this scenario really fail -- after the isolated retry has had its say? + * + * ONE definition, deliberately, because there are several consumers: the exit + * code, the flake count, and the comparison against the driver's + * expected-failure list. Written twice they would drift, and the drift would + * be silent in the worst possible place -- a row excused by one rule and + * counted by the other. + * + * A parallel-lane FAIL/ERROR that did not fail its isolated retry is a flake, + * not a failure. + */ +export declare function effectivelyFailed(verdict: Verdict, retryVerdict?: Verdict): boolean; +/** + * How an outcome lands against what the driver declared. This -- not the + * verdict -- is what the exit code reads. + * + * `expected-fail` is the point of the list: a finding already written down is + * not news, and excusing it one scenario at a time is what lets the other 46 + * be reported at all. + * + * `unexpected-pass` is what keeps the list from becoming the job-level mute it + * replaced. It fires on a declared scenario that did NOT effectively fail, + * which includes the case where it failed in parallel and passed its isolated + * retry -- there is no "expected flaky", so an entry that passes any way at + * all is an entry to look at. + * + * `unexpected-fail` covers BOTH an undeclared failure and a declared scenario + * that ERRORed. See {@link standingOf} for why the second belongs here. + */ +export type SweepStanding = "ok" | "expected-fail" | "unexpected-fail" | "unexpected-pass"; +/** + * A DECLARATION EXCUSES AN ANSWER, NEVER A CRASH. + * + * An expected-failure entry says what a CSMS *replies* -- its `reason` cites a + * handler, a status mapping, a field that comes back wrong. A scenario that + * ERRORs never got a reply to be wrong about: the container did not start, a + * bounded wait gave up, the driver threw. Treating that as the declared + * failure is how a job goes green on a breakage of a completely different + * nature from the one it documents -- which is the `continue-on-error` + * blindness this whole mechanism replaced, narrowed to one scenario but just + * as blind to the KIND of failure. + * + * So `expected-fail` needs the decisive verdict to be exactly FAIL. A declared + * scenario that errors is an ordinary unexpected failure, and the report says + * that its declaration is probably still good and the crash is the new thing. + * + * The full table, with `d` = decisive verdict ({@link decisiveVerdict}): + * + * | declared | effectively failed | d | standing | + * |----------|--------------------|--------|-----------------| + * | no | no | any | ok | + * | no | yes | any | unexpected-fail | + * | yes | yes | FAIL | expected-fail | + * | yes | yes | ERROR | unexpected-fail | + * | yes | no | any | unexpected-pass | + */ +export declare function standingOf(verdict: Verdict, expected: ExpectedFailureEntry | undefined, retryVerdict?: Verdict): SweepStanding; +/** + * Why an unexpected pass is one, in the words a reader needs to tell a fix + * from a flake from a contradiction from an unmeasured run -- without reading + * this file. + * + * The wording carries real weight, because the action it implies is DELETING A + * RECORDED FINDING. Only one of the cases below is actually evidence that the + * CSMS was fixed; saying so for the others would talk a maintainer into + * throwing away a finding that still holds. + */ +export declare function unexpectedPassDetail(verdict: Verdict, retryVerdict?: Verdict): string; +/** + * Why a DECLARED scenario is nevertheless an unexpected failure: it errored. + * + * Separated from the ordinary failure report because the two need opposite + * reactions. An undeclared failure is a finding to investigate and perhaps + * declare; this one almost certainly leaves the declaration intact and points + * at something new that stopped the scenario reaching the CSMS at all. + */ +export declare function declaredButErroredDetail(verdict: Verdict, retryVerdict?: Verdict): string; From a5cff4a8ac04f3000705dd380f1c18c5a41b8796 Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 14 Aug 2026 11:27:39 +0200 Subject: [PATCH 08/11] fix(tests): stop copying node_modules into the spec-invariant container `cp -r /src/. /work/` followed immediately by `rm -rf node_modules` carried ~640 files across the read-only bind mount for the sole purpose of deleting them, and on that mount the copy failed intermittently: busybox cp reporting "can't create link" for paths whose parent directory it had not created yet, roughly one run in three on macOS. A guard that goes red for a reason unrelated to the property it protects teaches people to re-run it, which is how it stops being read at all. Measured on origin/main before touching anything, so this predates any caller: clean checkout 6/6 green, the same checkout after `bun install` 5/6. After the change, 12/12 green here, and 12 runs take 13s where each one used to copy 638 files it did not want. find rather than a glob, because dotfiles must come across -- tsconfig.json sits beside .gitignore and a missing one changes what the extractors see -- and `/src/.[!.]*` matching nothing would expand to itself under `set -e`. Mutation-tested both directions: a hand-edited ASSERT-INVENTORY.txt line and a removed spec file each still fail, with their own diagnosis. Unrelated to the expected-failure work on this branch; it surfaced while running the gate for it. --- tests/spec-invariants.sh | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/tests/spec-invariants.sh b/tests/spec-invariants.sh index e328432..6bfebcf 100755 --- a/tests/spec-invariants.sh +++ b/tests/spec-invariants.sh @@ -65,15 +65,28 @@ trap 'rm -rf "$work"' EXIT INT TERM # The repository is mounted READ-ONLY and copied inside the # container before anything installs a dependency: no node_modules, lockfile or # generated artifact is ever written back into the repo by this test. +# +# node_modules is EXCLUDED FROM THE COPY rather than deleted after it, and that +# is a bug fix, not a speed-up. `cp -r /src/. /work/` followed by +# `rm -rf node_modules` copied ~640 files across the read-only bind mount for +# the sole purpose of deleting them, and on that mount the copy failed +# intermittently -- roughly one run in three on macOS -- with busybox cp +# reporting "can't create link" for paths whose parent directory it had not +# created yet. Measured on origin/main, so it predates any caller: clean +# checkout 6/6 green, same checkout after `bun install` 5/6. Never copying the +# tree is what removes the race; the guard's own inputs are 350 files. +# +# find, not a glob: dotfiles have to come across (tsconfig lives beside +# .gitignore, and a missing one changes what the extractors see), and +# `/src/.[!.]*` matching nothing would expand to itself under `set -e`. if ! docker run --rm \ -v "$repo_root:/src:ro" \ -v "$work:/out" \ "$bun_image" \ sh -c "set -e mkdir -p /work - cp -r /src/. /work/ + find /src -mindepth 1 -maxdepth 1 ! -name node_modules -exec cp -r {} /work/ ';' cd /work - rm -rf node_modules bun add -d 'typescript@$typescript_version' >/dev/null 2>&1 bun tools/extract-assert-inventory.ts tck/specs > /out/ASSERT-INVENTORY.txt bun tools/extract-drive-trace.ts > /out/DRIVE-TRACE.txt From 1e5d17694e4c37fa98a4bc582240f3a76cda9e41 Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 14 Aug 2026 12:46:33 +0200 Subject: [PATCH 09/11] fix(runner): let the compiler check that an excused row names its finding Review of the last three commits. Both findings are the same defect in two places: a declaration read through `?.` under a standing the compiler cannot tie back to it. Nothing renders "undefined" today; what is missing is anything stopping it. exitForSingleRun() reintroduced it. The nested ifs it replaced narrowed `expected` naturally, and switching on a SweepStanding -- a string -- does not, so the four field reads became optional chains. That is precisely the hazard declarationNote() was extracted to remove on the sweep side one commit earlier: a report that says a finding was excused while naming nothing is worse than no report, because it reads as reviewed. Undeclared now returns first, which narrows the rest. writeSummary()'s EXPECTED FAIL note had the same shape. withStanding() filters by standing AND narrows through a type predicate, so the three call sites that render `reason`/`finding` are checked rather than trusted; it also replaces three hand-written filters, one of which spelled the declared-but-errored predicate differently from its twin in runGroupSweep(). The single-scenario path also grows an explicit unexpected-fail branch. It is unreachable today -- a throw never reaches that function -- but folding it into a silent `return 1` would mean that the day the path learns to catch a throw, it exits 1 saying nothing about the declaration it was carrying. declaredButErroredDetail() was the one export of tck/standing.ts the guard did not assert. Its message points the reader AWAY from the entry, so inverting it would send them to delete a finding that still holds -- the same damage as a wrong UNEXPECTED PASS, from the other direction. Two assertions added, both mutation-tested to fail on their own wording. --- VENDOR.md | 2 +- patches/tck/main.ts.patch | 127 ++++++++++++++++++----------- tck/main.ts | 85 +++++++++++++------ tests/expected-failure-standing.ts | 17 ++++ 4 files changed, 159 insertions(+), 72 deletions(-) diff --git a/VENDOR.md b/VENDOR.md index 62a93c4..1d7e346 100644 --- a/VENDOR.md +++ b/VENDOR.md @@ -57,7 +57,7 @@ strongest available demonstration that the core names no CSMS. | `tck/specs/index.ts` | `upstream-verbatim` | `scripts/steve-verify/runner/specs/index.ts` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `—` | | `tck/assert.ts` | `upstream-patched` | `src/cp/application/verification/assert.ts` | `2431f5f6c0df997d4d821d9af55689c1f0f2df199de1e9e4ed6f3fbaad4fc89e` | `76d0f293db4f6ecc5768affea0ca76d2841c147fa3687dbea1f9e950bfbd9298` | `patches/tck/assert.ts.patch` | | `tck/sim.ts` | `upstream-patched` | `scripts/steve-verify/runner/sim.ts` | `2bf2f78afe3434e7139cd62c3ff6d70f02defd39dd700611e7c5f7614260cd35` | `45f897f273ea73227bb9b30766127eb9ecae0a319ab9048cbc30428fafff7f32` | `patches/tck/sim.ts.patch` | -| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `5febf8e25fd435d930798d2dd1d1771c448efb479d84fe9693a9aa27281f0aff` | `patches/tck/main.ts.patch` | +| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `399b6c7d2476884270f0bad6bb69344ace8045c8e299bffec621b116a9881868` | `patches/tck/main.ts.patch` | | `tck/driver.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/index.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/driver-registry.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | diff --git a/patches/tck/main.ts.patch b/patches/tck/main.ts.patch index 93ee1ba..907e6c2 100644 --- a/patches/tck/main.ts.patch +++ b/patches/tck/main.ts.patch @@ -228,8 +228,8 @@ + templateId: string, +): Promise { + return expectedFailureFor(await expectedFailureTable(), templateId); -+} -+ + } + +/** Result of one scenario execution -- either it produced checks, or the + * driver declared the scenario undrivable partway through drive(). */ +type ScenarioRun = @@ -267,8 +267,8 @@ +async function driverModule(): Promise { + driverModulePromise ??= loadDriverModule(); + return driverModulePromise; - } - ++} ++ +async function driver(): Promise { + driverPartsPromise ??= (async () => { + const module = await driverModule(); @@ -587,7 +587,7 @@ all: [ ...CORE_SPECS, ...AUTHLIST_RESERVATION_SPECS, -@@ -286,66 +529,154 @@ +@@ -286,66 +529,179 @@ .map((spec) => [spec.templateId, spec]), ); @@ -684,6 +684,10 @@ +function standingOfOutcome(outcome: ScenarioOutcome): SweepStanding { + return standingOf(outcome.verdict, outcome.expected, outcome.isolatedRetry?.verdict); +} ++ ++/** An outcome whose declaration is known present -- every standing except ++ * "ok" implies one, but a `SweepStanding` is a string and cannot say so. */ ++type DeclaredOutcome = ScenarioOutcome & { expected: ExpectedFailureEntry }; + /** - * Runs one scenario for a group sweep, isolating a thrown exception (e.g. @@ -692,6 +696,27 @@ - * sweep -- bash's per-scenario isolation comes for free from each - * run-scenario.sh invocation being a separate subprocess; this is the - * same-process equivalent. ++ * Outcomes with a given standing, NARROWED so their declaration can be read ++ * without `?.`. ++ * ++ * The narrowing is the point rather than the filtering. Reaching for ++ * `o.expected?.finding` inside a list already filtered to a declared standing ++ * compiles, and renders the literal "undefined" into a report if the standing ++ * rule and the declaration ever disagree -- which is a report saying a finding ++ * was excused, naming nothing. The predicate makes the compiler check what the ++ * filter already implies. + */ ++function withStanding( ++ outcomes: readonly ScenarioOutcome[], ++ standing: SweepStanding, ++): DeclaredOutcome[] { ++ return outcomes.filter( ++ (o): o is DeclaredOutcome => ++ o.expected !== undefined && standingOfOutcome(o) === standing, ++ ); ++} ++ ++/** + * The driver's declaration, as the one sentence every report renders -- + * `undefined` when it declared nothing about this scenario. + * @@ -703,7 +728,7 @@ + * Narrowing `expected` here also removes the `?.` chains the call sites needed: + * they could not see that a standing of `expected-fail` implies an entry, so a + * broken invariant would have rendered the string "undefined" into a report. - */ ++ */ +function declarationNote(outcome: ScenarioOutcome): string | undefined { + const { expected, verdict } = outcome; + const retryVerdict = outcome.isolatedRetry?.verdict; @@ -782,7 +807,7 @@ }; } catch (err) { const message = -@@ -353,64 +684,44 @@ +@@ -353,64 +709,44 @@ process.stderr.write( `[runner] ERROR: ${spec.templateId} on ${cpId} threw before completing: ${message}\n`, ); @@ -868,7 +893,7 @@ ); for (const outcome of toRetry) { -@@ -432,13 +743,14 @@ +@@ -432,13 +768,14 @@ verdict: retryOutcome.verdict, checks: retryOutcome.checks, failed: retryOutcome.failed, @@ -885,7 +910,7 @@ : " -- CONFIRMED (fails isolated too, not a parallel-lane artifact)\n"), ); } -@@ -451,11 +763,43 @@ +@@ -451,11 +788,43 @@ return new Date().toISOString().replace(/\.\d{3}Z$/, "Z"); } @@ -934,7 +959,7 @@ async function writeSummary( groupName: string, outcomes: ScenarioOutcome[], -@@ -465,45 +809,110 @@ +@@ -465,45 +834,104 @@ const rows = outcomes.map((o) => { const checks = o.checks === null ? "-" : String(o.checks); const failed = o.failed === null ? "-" : String(o.failed); @@ -980,12 +1005,8 @@ + `${naCount} NOT APPLICABLE (out of scope for this CSMS). Neither fails the sweep.`, + ); + } -+ const expectedFails = outcomes.filter( -+ (o) => standingOfOutcome(o) === "expected-fail", -+ ); -+ const unexpectedPasses = outcomes.filter( -+ (o) => standingOfOutcome(o) === "unexpected-pass", -+ ); ++ const expectedFails = withStanding(outcomes, "expected-fail"); ++ const unexpectedPasses = withStanding(outcomes, "unexpected-pass"); + if (expectedFails.length > 0) { + notes.push( + "", @@ -994,13 +1015,11 @@ + "rows stay DRIVABLE and the scenarios still ran; only the exit code " + + "changed. Each entry names where the finding is written down:", + ...expectedFails.map( -+ (o) => `- \`${o.templateId}\` — ${o.expected?.finding}`, ++ (o) => `- \`${o.templateId}\` — ${o.expected.finding}`, + ), + ); + } -+ const declaredButErrored = outcomes.filter( -+ (o) => o.expected !== undefined && standingOfOutcome(o) === "unexpected-fail", -+ ); ++ const declaredButErrored = withStanding(outcomes, "unexpected-fail"); + if (declaredButErrored.length > 0) { + notes.push( + "", @@ -1060,7 +1079,7 @@ "", header, separator, -@@ -518,54 +927,53 @@ +@@ -518,54 +946,53 @@ } /** Sequential or parallel group sweep. Writes results/summary.md and exits @@ -1140,7 +1159,7 @@ ); } else { for (let i = 0; i < specs.length; i++) { -@@ -574,11 +982,11 @@ +@@ -574,11 +1001,11 @@ } if (retryFailedIsolated) { @@ -1154,7 +1173,7 @@ "(a sequential sweep is already isolated).\n", ); } -@@ -588,45 +996,96 @@ +@@ -588,45 +1015,92 @@ for (const o of outcomes) { const retrySuffix = o.isolatedRetry ? ` [isolated retry: ${o.isolatedRetry.verdict}${ @@ -1185,18 +1204,14 @@ - (o) => o.verdict !== "PASS" && o.isolatedRetry?.verdict !== "PASS", + (o) => standingOfOutcome(o) === "unexpected-fail", ); -+ const expectedFails = outcomes.filter( -+ (o) => standingOfOutcome(o) === "expected-fail", -+ ); -+ const unexpectedPasses = outcomes.filter( -+ (o) => standingOfOutcome(o) === "unexpected-pass", -+ ); ++ const expectedFails = withStanding(outcomes, "expected-fail"); ++ const unexpectedPasses = withStanding(outcomes, "unexpected-pass"); + // A subset of badOutcomes, reported on its own line because it reads as a + // regression of the DECLARATION when it is nothing of the sort: the entry is + // almost certainly still good and something new stopped the scenario ever + // reaching the CSMS. Without this the maintainer sees a red build on the one + // row they were told would be red, and goes looking in the wrong place. -+ const declaredButErrored = badOutcomes.filter((o) => o.expected !== undefined); ++ const declaredButErrored = withStanding(outcomes, "unexpected-fail"); + // A flake IS "failed, but not effectively" -- said through the shared + // predicate rather than spelled out again, which is what the comment above + // claims and what the previous hand-written copy quietly contradicted. @@ -1266,7 +1281,7 @@ } // --------------------------------------------------------------------------- -@@ -642,6 +1101,8 @@ +@@ -642,6 +1116,8 @@ cpId: string; connector?: number; timeoutSecs?: number; @@ -1275,7 +1290,7 @@ } function requireValue(argv: string[], index: number, flag: string): string { -@@ -663,37 +1124,80 @@ +@@ -663,37 +1139,80 @@ return n; } @@ -1373,7 +1388,7 @@ if (argv[0] === "run-all") { group = "all"; -@@ -708,18 +1212,27 @@ +@@ -708,18 +1227,27 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1404,7 +1419,7 @@ process.exit(1); } -@@ -749,9 +1262,11 @@ +@@ -749,9 +1277,11 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1417,7 +1432,7 @@ process.exit(1); } } -@@ -764,35 +1279,495 @@ +@@ -764,35 +1294,495 @@ cpId, connector, timeoutSecs, @@ -1446,7 +1461,7 @@ + } + return [...seen].map(([templateId, group]) => ({ templateId, group })); +} - ++ +function listScenarios(argv: string[]): number { + let group: string | undefined; + let asJson = false; @@ -1629,7 +1644,7 @@ + ); + } + } -+ + + if (capabilities) { + const known = new Set(CSMS_OPERATION_ACTIONS); + const unknown = [...capabilities.operations].filter( @@ -1929,7 +1944,7 @@ } const spec = SPECS_BY_TEMPLATE_ID.get(args.templateId); -@@ -800,18 +1775,86 @@ +@@ -800,18 +1790,106 @@ process.stderr.write( `Unknown template id: ${args.templateId} (known: ${[...SPECS_BY_TEMPLATE_ID.keys()].join(", ")})\n`, ); @@ -2005,23 +2020,43 @@ + verdict: Verdict, + expected: ExpectedFailureEntry | undefined, +): number { -+ switch (standingOf(verdict, expected)) { -+ case "ok": -+ return 0; -+ case "unexpected-fail": -+ return 1; ++ const standing = standingOf(verdict, expected); ++ // Undeclared first, so `expected` is narrowed for the rest. Reading it ++ // through `?.` under a standing the compiler cannot tie back to it is how a ++ // report ends up saying a finding was excused while naming nothing -- see ++ // withStanding() for the same hazard on the sweep side. ++ if (expected === undefined) { ++ return standing === "unexpected-fail" ? 1 : 0; ++ } ++ switch (standing) { + case "expected-fail": + process.stderr.write( -+ `[runner] ${templateId} EXPECTED FAIL: ${expected?.reason} ` + -+ `(${expected?.finding}) -- declared by this driver, exit 0.\n`, ++ `[runner] ${templateId} EXPECTED FAIL: ${expected.reason} ` + ++ `(${expected.finding}) -- declared by this driver, exit 0.\n`, + ); + return 0; + case "unexpected-pass": + process.stderr.write( + `[runner] ${templateId} UNEXPECTED PASS: this driver declares it ` + -+ `expected-failing (${expected?.reason} -- ${expected?.finding}), ` + ++ `expected-failing (${expected.reason} -- ${expected.finding}), ` + + `and ${unexpectedPassDetail(verdict)}.\n`, + ); + return 1; ++ case "unexpected-fail": ++ // Not reachable today -- a throw never reaches this function, so ++ // `verdict` is never ERROR here and a declared FAIL is expected-fail. ++ // Written out anyway rather than folded into a default: "declared and ++ // errored" is the case the sweep exists to report, and the day this ++ // path learns to catch a throw it must not silently exit 1 with no word ++ // about the declaration it was carrying. ++ process.stderr.write( ++ `[runner] ${templateId} DECLARED, BUT ERRORED (${expected.reason} -- ` + ++ `${expected.finding}): ${declaredButErroredDetail(verdict)}.\n`, ++ ); ++ return 1; ++ case "ok": ++ // Unreachable: standingOf() answers "ok" only for an undeclared ++ // scenario, and that returned above. ++ return 0; + } +} diff --git a/tck/main.ts b/tck/main.ts index a420556..cb1c041 100644 --- a/tck/main.ts +++ b/tck/main.ts @@ -595,6 +595,31 @@ function standingOfOutcome(outcome: ScenarioOutcome): SweepStanding { return standingOf(outcome.verdict, outcome.expected, outcome.isolatedRetry?.verdict); } +/** An outcome whose declaration is known present -- every standing except + * "ok" implies one, but a `SweepStanding` is a string and cannot say so. */ +type DeclaredOutcome = ScenarioOutcome & { expected: ExpectedFailureEntry }; + +/** + * Outcomes with a given standing, NARROWED so their declaration can be read + * without `?.`. + * + * The narrowing is the point rather than the filtering. Reaching for + * `o.expected?.finding` inside a list already filtered to a declared standing + * compiles, and renders the literal "undefined" into a report if the standing + * rule and the declaration ever disagree -- which is a report saying a finding + * was excused, naming nothing. The predicate makes the compiler check what the + * filter already implies. + */ +function withStanding( + outcomes: readonly ScenarioOutcome[], + standing: SweepStanding, +): DeclaredOutcome[] { + return outcomes.filter( + (o): o is DeclaredOutcome => + o.expected !== undefined && standingOfOutcome(o) === standing, + ); +} + /** * The driver's declaration, as the one sentence every report renders -- * `undefined` when it declared nothing about this scenario. @@ -845,12 +870,8 @@ async function writeSummary( `${naCount} NOT APPLICABLE (out of scope for this CSMS). Neither fails the sweep.`, ); } - const expectedFails = outcomes.filter( - (o) => standingOfOutcome(o) === "expected-fail", - ); - const unexpectedPasses = outcomes.filter( - (o) => standingOfOutcome(o) === "unexpected-pass", - ); + const expectedFails = withStanding(outcomes, "expected-fail"); + const unexpectedPasses = withStanding(outcomes, "unexpected-pass"); if (expectedFails.length > 0) { notes.push( "", @@ -859,13 +880,11 @@ async function writeSummary( "rows stay DRIVABLE and the scenarios still ran; only the exit code " + "changed. Each entry names where the finding is written down:", ...expectedFails.map( - (o) => `- \`${o.templateId}\` — ${o.expected?.finding}`, + (o) => `- \`${o.templateId}\` — ${o.expected.finding}`, ), ); } - const declaredButErrored = outcomes.filter( - (o) => o.expected !== undefined && standingOfOutcome(o) === "unexpected-fail", - ); + const declaredButErrored = withStanding(outcomes, "unexpected-fail"); if (declaredButErrored.length > 0) { notes.push( "", @@ -1016,18 +1035,14 @@ async function runGroupSweep( const badOutcomes = outcomes.filter( (o) => standingOfOutcome(o) === "unexpected-fail", ); - const expectedFails = outcomes.filter( - (o) => standingOfOutcome(o) === "expected-fail", - ); - const unexpectedPasses = outcomes.filter( - (o) => standingOfOutcome(o) === "unexpected-pass", - ); + const expectedFails = withStanding(outcomes, "expected-fail"); + const unexpectedPasses = withStanding(outcomes, "unexpected-pass"); // A subset of badOutcomes, reported on its own line because it reads as a // regression of the DECLARATION when it is nothing of the sort: the entry is // almost certainly still good and something new stopped the scenario ever // reaching the CSMS. Without this the maintainer sees a red build on the one // row they were told would be red, and goes looking in the wrong place. - const declaredButErrored = badOutcomes.filter((o) => o.expected !== undefined); + const declaredButErrored = withStanding(outcomes, "unexpected-fail"); // A flake IS "failed, but not effectively" -- said through the shared // predicate rather than spelled out again, which is what the comment above // claims and what the previous hand-written copy quietly contradicted. @@ -1838,23 +1853,43 @@ function exitForSingleRun( verdict: Verdict, expected: ExpectedFailureEntry | undefined, ): number { - switch (standingOf(verdict, expected)) { - case "ok": - return 0; - case "unexpected-fail": - return 1; + const standing = standingOf(verdict, expected); + // Undeclared first, so `expected` is narrowed for the rest. Reading it + // through `?.` under a standing the compiler cannot tie back to it is how a + // report ends up saying a finding was excused while naming nothing -- see + // withStanding() for the same hazard on the sweep side. + if (expected === undefined) { + return standing === "unexpected-fail" ? 1 : 0; + } + switch (standing) { case "expected-fail": process.stderr.write( - `[runner] ${templateId} EXPECTED FAIL: ${expected?.reason} ` + - `(${expected?.finding}) -- declared by this driver, exit 0.\n`, + `[runner] ${templateId} EXPECTED FAIL: ${expected.reason} ` + + `(${expected.finding}) -- declared by this driver, exit 0.\n`, ); return 0; case "unexpected-pass": process.stderr.write( `[runner] ${templateId} UNEXPECTED PASS: this driver declares it ` + - `expected-failing (${expected?.reason} -- ${expected?.finding}), ` + + `expected-failing (${expected.reason} -- ${expected.finding}), ` + `and ${unexpectedPassDetail(verdict)}.\n`, ); return 1; + case "unexpected-fail": + // Not reachable today -- a throw never reaches this function, so + // `verdict` is never ERROR here and a declared FAIL is expected-fail. + // Written out anyway rather than folded into a default: "declared and + // errored" is the case the sweep exists to report, and the day this + // path learns to catch a throw it must not silently exit 1 with no word + // about the declaration it was carrying. + process.stderr.write( + `[runner] ${templateId} DECLARED, BUT ERRORED (${expected.reason} -- ` + + `${expected.finding}): ${declaredButErroredDetail(verdict)}.\n`, + ); + return 1; + case "ok": + // Unreachable: standingOf() answers "ok" only for an undeclared + // scenario, and that returned above. + return 0; } } diff --git a/tests/expected-failure-standing.ts b/tests/expected-failure-standing.ts index da99e60..1321a2a 100644 --- a/tests/expected-failure-standing.ts +++ b/tests/expected-failure-standing.ts @@ -26,6 +26,7 @@ */ import type { ExpectedFailureEntry } from "../tck/expected"; import { + declaredButErroredDetail, effectivelyFailed, standingOf, unexpectedPassDetail, @@ -123,6 +124,22 @@ check( "is the one case where it is.", ); +// The mirror of the three above. This message points the reader AWAY from the +// entry -- the declaration is probably intact and the crash is new -- so +// inverting it would send them to delete a finding that still holds, which is +// the same damage from the opposite direction. +check( + declaredButErroredDetail("ERROR").includes("still good"), + "a declared scenario that errored no longer reads as 'the entry is " + + "probably still good', so the message now points at the declaration " + + "rather than at the crash.", +); +check( + declaredButErroredDetail("FAIL", "ERROR").includes("isolated retry"), + "the FAIL-then-ERROR case no longer says where the ERROR came from, so a " + + "reader cannot tell a crashing retry from a crashing lane.", +); + function check(condition: boolean, failure: string): void { if (!condition) failures.push(failure); } From 94b4ffb28a5e504190572b998dcf6a34f01dccf7 Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 14 Aug 2026 13:02:06 +0200 Subject: [PATCH 10/11] refactor(runner): give DECLARED, BUT ERRORED a name in the type Cleanup pass. No behaviour change: the same verdicts produce the same exit codes and the same reports, checked on the sweep and single-run paths. README.md and CONTRIBUTING.md both name DECLARED, BUT ERRORED, the sweep reports it on its own line, and it called for the opposite reaction to an ordinary failure -- keep the entry, chase the crash. But SweepStanding had four members, so it existed only as "unexpected-fail AND a declaration is present", a conjunct hiding inside a filter. Two call sites ten lines apart asked the same-looking question and got different sets, and withStanding()'s own doc claimed something false: "every standing except ok implies a declaration" -- unexpected-fail does not, which is exactly why badOutcomes could not use the helper. It is now a member. DeclaredStanding is the sub-union that does imply a declaration, so withStanding() takes only those and the promise is checked at each call site rather than asserted in prose. endsTheBuild() owns the exit disjunction so a sixth standing cannot be added and left out of it. Both exhaustive switches now name the case instead of commenting it -- and the compiler found them the moment the member landed, which is the argument for the member. partitionByStanding() computes every bucket once. writeSummary() and runGroupSweep() each rebuilt the same three-line block, and the flake predicate was still spelled by hand in two more places, under a comment claiming it had been unified. effectivelyFailed() now reads through decisiveVerdict(), which gives that function a second caller and makes the doc table's two columns visibly the same quantity. THE GUARD'S PROSE ASSERTIONS ARE REPLACED, because they were shown not to work. Grepping the messages for "looks fixed" is brittle to a harmless reword AND blind to the harmful one: rewording the PARTIAL branch to "the finding looks resolved" -- the maintainer-misleading defect they existed to catch -- left them green. unexpectedPassKind() exposes the distinction the rule already computes, so the guard pins a value table instead of a token, and keeps one prose property that names no phrase: the kinds must stay tellable apart. Mutation-tested, including that planted reword, which now fails. Also excludes .git from the spec-invariant container copy. Neither extractor reads it, and at ~1600 files in a normal clone it is several times the cost of everything the copy legitimately carries -- each one another draw in the busybox-cp lottery the node_modules exclusion was fixing. 6/6 green after. --- VENDOR.md | 2 +- patches/tck/main.ts.patch | 243 ++++++++++++++++++----------- tck/main.ts | 130 +++++++++------ tck/standing.ts | 110 +++++++++---- tests/expected-failure-standing.ts | 96 ++++++++---- tests/spec-invariants.sh | 13 +- types/tck/standing.d.ts | 46 +++++- 7 files changed, 423 insertions(+), 217 deletions(-) diff --git a/VENDOR.md b/VENDOR.md index 1d7e346..0cca3aa 100644 --- a/VENDOR.md +++ b/VENDOR.md @@ -57,7 +57,7 @@ strongest available demonstration that the core names no CSMS. | `tck/specs/index.ts` | `upstream-verbatim` | `scripts/steve-verify/runner/specs/index.ts` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `—` | | `tck/assert.ts` | `upstream-patched` | `src/cp/application/verification/assert.ts` | `2431f5f6c0df997d4d821d9af55689c1f0f2df199de1e9e4ed6f3fbaad4fc89e` | `76d0f293db4f6ecc5768affea0ca76d2841c147fa3687dbea1f9e950bfbd9298` | `patches/tck/assert.ts.patch` | | `tck/sim.ts` | `upstream-patched` | `scripts/steve-verify/runner/sim.ts` | `2bf2f78afe3434e7139cd62c3ff6d70f02defd39dd700611e7c5f7614260cd35` | `45f897f273ea73227bb9b30766127eb9ecae0a319ab9048cbc30428fafff7f32` | `patches/tck/sim.ts.patch` | -| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `399b6c7d2476884270f0bad6bb69344ace8045c8e299bffec621b116a9881868` | `patches/tck/main.ts.patch` | +| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `aa1153bc54894707510654d818382b1169406a2d2927822ac018c2e470fa21a1` | `patches/tck/main.ts.patch` | | `tck/driver.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/index.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/driver-registry.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | diff --git a/patches/tck/main.ts.patch b/patches/tck/main.ts.patch index 907e6c2..6b10af7 100644 --- a/patches/tck/main.ts.patch +++ b/patches/tck/main.ts.patch @@ -1,6 +1,6 @@ --- /tmp/ocpp-upstream/scripts/steve-verify/runner/main.ts 2026-07-31 20:05:48 +++ tck/main.ts 2026-07-31 20:12:10 -@@ -1,39 +1,95 @@ +@@ -1,39 +1,97 @@ #!/usr/bin/env bun +// Derived from shiv3/ocpp-cp-simulator scripts/steve-verify/runner/main.ts @ 604054adb0d7d7129a26a5f1ad2d5fdc290d1ca1 (Apache-2.0). Modified: the STEVE_DRIVER=api|ui selection is replaced by a CSMS driver loaded through ./driver-registry; a per-driver scope table (./scope) is consulted BEFORE any container starts and yields the NOT APPLICABLE verdict; UnsupportedOperationError (./driver) thrown out of drive() degrades to NOT APPLICABLE with a stderr WARNING; the PARTIAL verdict and the `skipped` summary column were added; the exit code is non-zero only for FAIL/ERROR; parallel lanes derive from the resolved station list instead of the fixed CERTCP1..3 trio; the SteVe capability probe is dropped. + @@ -86,9 +86,11 @@ +import { + declaredButErroredDetail, + effectivelyFailed, ++ endsTheBuild, + isFailure, + standingOf, + unexpectedPassDetail, ++ type DeclaredStanding, + type SweepStanding, + type Verdict, +} from "./standing"; @@ -122,7 +124,7 @@ import { AUTHLIST_RESERVATION_SPECS, AUTHORIZE_SPECS, -@@ -43,13 +99,25 @@ +@@ -43,13 +101,25 @@ } from "./specs/index"; import type { ScenarioSpec } from "./spec-types"; import { sleep } from "./util"; @@ -153,7 +155,7 @@ // --------------------------------------------------------------------------- // Runner core // --------------------------------------------------------------------------- -@@ -60,95 +128,234 @@ +@@ -60,95 +130,234 @@ timeoutSecs?: number; } @@ -228,8 +230,8 @@ + templateId: string, +): Promise { + return expectedFailureFor(await expectedFailureTable(), templateId); - } - ++} ++ +/** Result of one scenario execution -- either it produced checks, or the + * driver declared the scenario undrivable partway through drive(). */ +type ScenarioRun = @@ -267,8 +269,8 @@ +async function driverModule(): Promise { + driverModulePromise ??= loadDriverModule(); + return driverModulePromise; -+} -+ + } + +async function driver(): Promise { + driverPartsPromise ??= (async () => { + const module = await driverModule(); @@ -449,7 +451,7 @@ 30_000, ); } catch (err) { -@@ -176,13 +383,25 @@ +@@ -176,13 +385,25 @@ if (spec.drive) { process.stderr.write(`[runner] running drive() for ${spec.templateId}\n`); @@ -482,7 +484,7 @@ } else { process.stderr.write( `[runner] no drive() defined (CP-only scenario) -- nothing to do while it runs\n`, -@@ -190,15 +409,18 @@ +@@ -190,15 +411,18 @@ driveState = undefined as D; } @@ -504,7 +506,7 @@ // it a FAIL in a swept scenario is un-post-mortem-able: the sim container // is already stopped+removed by the time assert() reports, so this // capture is the only surviving record of what was (or wasn't) on the -@@ -218,6 +440,10 @@ +@@ -218,6 +442,10 @@ ); } @@ -515,7 +517,7 @@ const frames = parseLog(lines.join("\n")); const rec = new AssertRecorder(); -@@ -228,34 +454,53 @@ +@@ -228,34 +456,53 @@ frames, lines, rec, @@ -578,7 +580,7 @@ // --------------------------------------------------------------------------- // eslint-disable-next-line @typescript-eslint/no-explicit-any -@@ -265,8 +510,6 @@ +@@ -265,8 +512,6 @@ "remotetrigger-smartcharging": REMOTETRIGGER_SMARTCHARGING_SPECS, firmware: FIRMWARE_SPECS, authorize: AUTHORIZE_SPECS, @@ -587,7 +589,7 @@ all: [ ...CORE_SPECS, ...AUTHLIST_RESERVATION_SPECS, -@@ -286,66 +529,179 @@ +@@ -286,66 +531,227 @@ .map((spec) => [spec.templateId, spec]), ); @@ -670,14 +672,13 @@ - checks: number | null; - failed: number | null; - errorMessage?: string; -- }; + isolatedRetry?: RetryOutcome; + /** The driver's declaration that this scenario is known to fail, attached + * whether or not it did. Both directions are reported: a declared FAIL is + * excused, a declared PASS fails the sweep. */ + expected?: ExpectedFailureEntry; - } - ++} ++ +/** {@link standingOf} for a sweep outcome. The single-scenario path in `cli()` + * has no ScenarioOutcome and calls the fields form directly, which is the + * point of the split: the two paths cannot decide differently. */ @@ -689,26 +690,24 @@ + * "ok" implies one, but a `SweepStanding` is a string and cannot say so. */ +type DeclaredOutcome = ScenarioOutcome & { expected: ExpectedFailureEntry }; + - /** -- * Runs one scenario for a group sweep, isolating a thrown exception (e.g. -- * waitForCondition/waitActiveTxPk's fail-hard timeout rejection propagating -- * out of drive()) to THIS scenario's outcome instead of aborting the whole -- * sweep -- bash's per-scenario isolation comes for free from each -- * run-scenario.sh invocation being a separate subprocess; this is the -- * same-process equivalent. -+ * Outcomes with a given standing, NARROWED so their declaration can be read -+ * without `?.`. ++/** ++ * Outcomes with a given DECLARED standing, narrowed so the declaration can be ++ * read without `?.`. + * + * The narrowing is the point rather than the filtering. Reaching for + * `o.expected?.finding` inside a list already filtered to a declared standing + * compiles, and renders the literal "undefined" into a report if the standing -+ * rule and the declaration ever disagree -- which is a report saying a finding -+ * was excused, naming nothing. The predicate makes the compiler check what the -+ * filter already implies. - */ ++ * rule and the declaration ever disagree -- a report saying a finding was ++ * excused while naming nothing. ++ * ++ * The parameter is {@link DeclaredStanding}, not SweepStanding, and that is ++ * what makes the promise honest rather than merely asserted: an undeclared ++ * standing cannot be passed, so "this standing implies a declaration" is ++ * checked at every call site instead of stated in this comment. ++ */ +function withStanding( + outcomes: readonly ScenarioOutcome[], -+ standing: SweepStanding, ++ standing: DeclaredStanding, +): DeclaredOutcome[] { + return outcomes.filter( + (o): o is DeclaredOutcome => @@ -717,23 +716,71 @@ +} + +/** ++ * Every bucket the sweep reports, in ONE pass and ONE definition. ++ * ++ * Both the markdown summary and the stderr conclusion partition the same ++ * outcomes the same way, and they used to do it with two identical-looking ++ * blocks in different functions -- the drift hazard `effectivelyFailed`'s own ++ * doc warns about, one level up. `flakes` is here for the same reason: it was ++ * spelled out by hand in three places, twice differently. ++ */ ++interface StandingPartition { ++ unexpectedFails: ScenarioOutcome[]; ++ expectedFails: DeclaredOutcome[]; ++ declaredButErrored: DeclaredOutcome[]; ++ unexpectedPasses: DeclaredOutcome[]; ++ /** Failed in a lane, did not fail isolated. Not a failure; reported so a ++ * red-looking run explains itself. */ ++ flakes: ScenarioOutcome[]; ++} ++ ++function partitionByStanding( ++ outcomes: readonly ScenarioOutcome[], ++): StandingPartition { ++ return { ++ unexpectedFails: outcomes.filter( ++ (o) => standingOfOutcome(o) === "unexpected-fail", ++ ), ++ expectedFails: withStanding(outcomes, "expected-fail"), ++ declaredButErrored: withStanding(outcomes, "declared-but-errored"), ++ unexpectedPasses: withStanding(outcomes, "unexpected-pass"), ++ flakes: outcomes.filter( ++ (o) => ++ isFailure(o.verdict) && ++ !effectivelyFailed(o.verdict, o.isolatedRetry?.verdict), ++ ), + }; + } + + /** +- * Runs one scenario for a group sweep, isolating a thrown exception (e.g. +- * waitForCondition/waitActiveTxPk's fail-hard timeout rejection propagating +- * out of drive()) to THIS scenario's outcome instead of aborting the whole +- * sweep -- bash's per-scenario isolation comes for free from each +- * run-scenario.sh invocation being a separate subprocess; this is the +- * same-process equivalent. + * The driver's declaration, as the one sentence every report renders -- + * `undefined` when it declared nothing about this scenario. + * -+ * Written once because it was written three times: the markdown summary, the -+ * sweep's stderr list and the single-scenario line each rebuilt it, and the -+ * fourth copy had already drifted to a different separator. It is the same -+ * mistake unexpectedPassDetail() was extracted to fix, one level up. ++ * Written once because it was written three times, and one copy had already ++ * drifted to a different separator. The two sweep renderers -- the markdown ++ * verdict cell and the stderr list -- go through here. exitForSingleRun() ++ * deliberately does NOT: its sentences have to explain an exit code and ++ * interleave the declaration differently, so forcing them through one renderer ++ * would cost more than the labels it shares. + * + * Narrowing `expected` here also removes the `?.` chains the call sites needed: + * they could not see that a standing of `expected-fail` implies an entry, so a + * broken invariant would have rendered the string "undefined" into a report. -+ */ + */ +function declarationNote(outcome: ScenarioOutcome): string | undefined { + const { expected, verdict } = outcome; + const retryVerdict = outcome.isolatedRetry?.verdict; + if (!expected) return undefined; -+ switch (standingOf(verdict, expected, retryVerdict)) { ++ // standingOfOutcome(), not standingOf() on unpacked fields: unpacking here ++ // would be a third place that maps an outcome to a standing, and the exit ++ // code reads the other one. ++ switch (standingOfOutcome(outcome)) { + case "expected-fail": + return `EXPECTED FAIL: ${expected.reason} (${expected.finding})`; + case "unexpected-pass": @@ -741,14 +788,16 @@ + "UNEXPECTED PASS: declared expected-failing, but " + + unexpectedPassDetail(verdict, retryVerdict) + ); -+ case "unexpected-fail": ++ case "declared-but-errored": + return ( + `DECLARED, BUT ERRORED (${expected.reason} -- ${expected.finding}): ` + + declaredButErroredDetail(verdict, retryVerdict) + ); + case "ok": -+ // Unreachable: `expected` is non-undefined here, and standingOf() only -+ // answers "ok" for an undeclared scenario. ++ case "unexpected-fail": ++ // Unreachable: `expected` is non-undefined here, and both of these are ++ // the undeclared standings. There is no note to write for a scenario ++ // this driver said nothing about. + return undefined; + } +} @@ -807,7 +856,7 @@ }; } catch (err) { const message = -@@ -353,64 +709,44 @@ +@@ -353,64 +759,44 @@ process.stderr.write( `[runner] ERROR: ${spec.templateId} on ${cpId} threw before completing: ${message}\n`, ); @@ -893,7 +942,7 @@ ); for (const outcome of toRetry) { -@@ -432,13 +768,14 @@ +@@ -432,13 +818,14 @@ verdict: retryOutcome.verdict, checks: retryOutcome.checks, failed: retryOutcome.failed, @@ -910,7 +959,7 @@ : " -- CONFIRMED (fails isolated too, not a parallel-lane artifact)\n"), ); } -@@ -451,11 +788,43 @@ +@@ -451,59 +838,147 @@ return new Date().toISOString().replace(/\.\d{3}Z$/, "Z"); } @@ -959,7 +1008,10 @@ async function writeSummary( groupName: string, outcomes: ScenarioOutcome[], -@@ -465,45 +834,104 @@ ++ parts: StandingPartition, + ): Promise { + const anyRetried = outcomes.some((o) => o.isolatedRetry !== undefined); + const rows = outcomes.map((o) => { const checks = o.checks === null ? "-" : String(o.checks); const failed = o.failed === null ? "-" : String(o.failed); @@ -994,6 +1046,10 @@ + : "| --- | --- | --- | --- | --- | --- |"; const notes: string[] = []; +- if (anyRetried) { +- const flakeCount = outcomes.filter( +- (o) => o.isolatedRetry?.verdict === "PASS", +- ).length; + const partialCount = outcomes.filter((o) => o.verdict === "PARTIAL").length; + const naCount = outcomes.filter( + (o) => o.verdict === "NOT APPLICABLE", @@ -1005,8 +1061,7 @@ + `${naCount} NOT APPLICABLE (out of scope for this CSMS). Neither fails the sweep.`, + ); + } -+ const expectedFails = withStanding(outcomes, "expected-fail"); -+ const unexpectedPasses = withStanding(outcomes, "unexpected-pass"); ++ const { expectedFails, unexpectedPasses, declaredButErrored } = parts; + if (expectedFails.length > 0) { + notes.push( + "", @@ -1019,7 +1074,6 @@ + ), + ); + } -+ const declaredButErrored = withStanding(outcomes, "unexpected-fail"); + if (declaredButErrored.length > 0) { + notes.push( + "", @@ -1047,11 +1101,8 @@ + ), + ); + } - if (anyRetried) { - const flakeCount = outcomes.filter( -- (o) => o.isolatedRetry?.verdict === "PASS", -+ (o) => o.isolatedRetry !== undefined && !isFailure(o.isolatedRetry.verdict), - ).length; ++ if (anyRetried) { ++ const flakeCount = parts.flakes.length; const confirmedCount = outcomes.filter( - (o) => - o.isolatedRetry !== undefined && o.isolatedRetry.verdict !== "PASS", @@ -1079,7 +1130,7 @@ "", header, separator, -@@ -518,54 +946,53 @@ +@@ -518,54 +993,53 @@ } /** Sequential or parallel group sweep. Writes results/summary.md and exits @@ -1159,7 +1210,7 @@ ); } else { for (let i = 0; i < specs.length; i++) { -@@ -574,11 +1001,11 @@ +@@ -574,11 +1048,11 @@ } if (retryFailedIsolated) { @@ -1173,7 +1224,7 @@ "(a sequential sweep is already isolated).\n", ); } -@@ -588,45 +1015,92 @@ +@@ -588,45 +1062,86 @@ for (const o of outcomes) { const retrySuffix = o.isolatedRetry ? ` [isolated retry: ${o.isolatedRetry.verdict}${ @@ -1190,46 +1241,46 @@ ); } - const summaryPath = await writeSummary(groupName, outcomes); +- const summaryPath = await writeSummary(groupName, outcomes); ++ const parts = partitionByStanding(outcomes); ++ const summaryPath = await writeSummary(groupName, outcomes, parts); process.stderr.write(`[runner] results table: ${summaryPath}\n`); - // A parallel-lane FAIL/ERROR that PASSed on its isolated retry is a flake - // (issue #184 Finding 4), not a real failure -- it does not fail the - // sweep. Anything else non-PASS (no retry attempted, or still non-PASS - // isolated) counts as a real failure. +- const badOutcomes = outcomes.filter( +- (o) => o.verdict !== "PASS" && o.isolatedRetry?.verdict !== "PASS", +- ); +- const flakeCount = outcomes.filter( +- (o) => o.verdict !== "PASS" && o.isolatedRetry?.verdict === "PASS", +- ).length; + // A parallel-lane FAIL/ERROR that did not fail on its isolated retry is a + // flake, not a real failure -- it does not fail the sweep. Everything below + // reads that through effectivelyFailed(), via standingOf(). - const badOutcomes = outcomes.filter( -- (o) => o.verdict !== "PASS" && o.isolatedRetry?.verdict !== "PASS", -+ (o) => standingOfOutcome(o) === "unexpected-fail", - ); -+ const expectedFails = withStanding(outcomes, "expected-fail"); -+ const unexpectedPasses = withStanding(outcomes, "unexpected-pass"); ++ const { unexpectedFails, expectedFails, unexpectedPasses, declaredButErrored, flakes } = ++ parts; + // A subset of badOutcomes, reported on its own line because it reads as a + // regression of the DECLARATION when it is nothing of the sort: the entry is + // almost certainly still good and something new stopped the scenario ever + // reaching the CSMS. Without this the maintainer sees a red build on the one + // row they were told would be red, and goes looking in the wrong place. -+ const declaredButErrored = withStanding(outcomes, "unexpected-fail"); -+ // A flake IS "failed, but not effectively" -- said through the shared -+ // predicate rather than spelled out again, which is what the comment above -+ // claims and what the previous hand-written copy quietly contradicted. - const flakeCount = outcomes.filter( -- (o) => o.verdict !== "PASS" && o.isolatedRetry?.verdict === "PASS", -+ (o) => isFailure(o.verdict) && !effectivelyFailed(o.verdict, o.isolatedRetry?.verdict), - ).length; ++ const flakeCount = flakes.length; if (flakeCount > 0) { process.stderr.write( - `[runner] ${flakeCount} parallel-only flake(s) in group '${groupName}' (PASSed on isolated retry) -- see ${summaryPath}\n`, + `[runner] ${flakeCount} parallel-only flake(s) in group '${groupName}' (did not fail on isolated retry) -- see ${summaryPath}\n`, ); } +- if (badOutcomes.length > 0) { + if (expectedFails.length > 0) { -+ process.stderr.write( + process.stderr.write( +- `[runner] ${badOutcomes.length}/${outcomes.length} scenario(s) in group '${groupName}' FAILed or errored -- see ${summaryPath}\n`, + `[runner] ${expectedFails.length} scenario(s) in group '${groupName}' failed as this driver declared they would -- not a build failure; see ${summaryPath}\n`, -+ ); -+ } + ); +- process.exit(1); + } + if (declaredButErrored.length > 0) { + process.stderr.write( + `[runner] ${declaredButErrored.length} scenario(s) in group '${groupName}' are declared expected-failing and ERRORED, which is not the declared failure:\n`, @@ -1253,13 +1304,14 @@ + ); + } + } - if (badOutcomes.length > 0) { - process.stderr.write( - `[runner] ${badOutcomes.length}/${outcomes.length} scenario(s) in group '${groupName}' FAILed or errored -- see ${summaryPath}\n`, - ); -- process.exit(1); - } -+ if (badOutcomes.length > 0 || unexpectedPasses.length > 0) return 1; ++ if (unexpectedFails.length > 0) { ++ process.stderr.write( ++ `[runner] ${unexpectedFails.length}/${outcomes.length} scenario(s) in group '${groupName}' FAILed or errored -- see ${summaryPath}\n`, ++ ); ++ } ++ // The disjunction lives in standing.ts, so a sixth standing cannot be added ++ // and silently left out of the exit code here. ++ if (outcomes.some((o) => endsTheBuild(standingOfOutcome(o)))) return 1; + + const partialCount = outcomes.filter((o) => o.verdict === "PARTIAL").length; + const naCount = outcomes.filter((o) => o.verdict === "NOT APPLICABLE").length; @@ -1281,7 +1333,7 @@ } // --------------------------------------------------------------------------- -@@ -642,6 +1116,8 @@ +@@ -642,6 +1157,8 @@ cpId: string; connector?: number; timeoutSecs?: number; @@ -1290,7 +1342,7 @@ } function requireValue(argv: string[], index: number, flag: string): string { -@@ -663,37 +1139,80 @@ +@@ -663,37 +1180,80 @@ return n; } @@ -1388,7 +1440,7 @@ if (argv[0] === "run-all") { group = "all"; -@@ -708,18 +1227,27 @@ +@@ -708,18 +1268,27 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1419,7 +1471,7 @@ process.exit(1); } -@@ -749,9 +1277,11 @@ +@@ -749,9 +1318,11 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1432,7 +1484,7 @@ process.exit(1); } } -@@ -764,35 +1294,495 @@ +@@ -764,35 +1335,495 @@ cpId, connector, timeoutSecs, @@ -1461,7 +1513,7 @@ + } + return [...seen].map(([templateId, group]) => ({ templateId, group })); +} -+ + +function listScenarios(argv: string[]): number { + let group: string | undefined; + let asJson = false; @@ -1644,7 +1696,7 @@ + ); + } + } - ++ + if (capabilities) { + const known = new Set(CSMS_OPERATION_ACTIONS); + const unknown = [...capabilities.operations].filter( @@ -1944,7 +1996,7 @@ } const spec = SPECS_BY_TEMPLATE_ID.get(args.templateId); -@@ -800,18 +1790,106 @@ +@@ -800,18 +1831,107 @@ process.stderr.write( `Unknown template id: ${args.templateId} (known: ${[...SPECS_BY_TEMPLATE_ID.keys()].join(", ")})\n`, ); @@ -2042,21 +2094,22 @@ + `and ${unexpectedPassDetail(verdict)}.\n`, + ); + return 1; -+ case "unexpected-fail": ++ case "declared-but-errored": + // Not reachable today -- a throw never reaches this function, so + // `verdict` is never ERROR here and a declared FAIL is expected-fail. -+ // Written out anyway rather than folded into a default: "declared and -+ // errored" is the case the sweep exists to report, and the day this -+ // path learns to catch a throw it must not silently exit 1 with no word -+ // about the declaration it was carrying. ++ // Written out anyway rather than folded into a default: this is the case ++ // the sweep exists to report, and the day this path learns to catch a ++ // throw it must not silently exit 1 with no word about the declaration ++ // it was carrying. + process.stderr.write( + `[runner] ${templateId} DECLARED, BUT ERRORED (${expected.reason} -- ` + + `${expected.finding}): ${declaredButErroredDetail(verdict)}.\n`, + ); + return 1; + case "ok": -+ // Unreachable: standingOf() answers "ok" only for an undeclared -+ // scenario, and that returned above. -+ return 0; ++ case "unexpected-fail": ++ // Unreachable: both are the undeclared standings, and undeclared ++ // returned above. ++ return endsTheBuild(standing) ? 1 : 0; + } +} diff --git a/tck/main.ts b/tck/main.ts index cb1c041..a428347 100644 --- a/tck/main.ts +++ b/tck/main.ts @@ -65,9 +65,11 @@ import { import { declaredButErroredDetail, effectivelyFailed, + endsTheBuild, isFailure, standingOf, unexpectedPassDetail, + type DeclaredStanding, type SweepStanding, type Verdict, } from "./standing"; @@ -600,19 +602,23 @@ function standingOfOutcome(outcome: ScenarioOutcome): SweepStanding { type DeclaredOutcome = ScenarioOutcome & { expected: ExpectedFailureEntry }; /** - * Outcomes with a given standing, NARROWED so their declaration can be read - * without `?.`. + * Outcomes with a given DECLARED standing, narrowed so the declaration can be + * read without `?.`. * * The narrowing is the point rather than the filtering. Reaching for * `o.expected?.finding` inside a list already filtered to a declared standing * compiles, and renders the literal "undefined" into a report if the standing - * rule and the declaration ever disagree -- which is a report saying a finding - * was excused, naming nothing. The predicate makes the compiler check what the - * filter already implies. + * rule and the declaration ever disagree -- a report saying a finding was + * excused while naming nothing. + * + * The parameter is {@link DeclaredStanding}, not SweepStanding, and that is + * what makes the promise honest rather than merely asserted: an undeclared + * standing cannot be passed, so "this standing implies a declaration" is + * checked at every call site instead of stated in this comment. */ function withStanding( outcomes: readonly ScenarioOutcome[], - standing: SweepStanding, + standing: DeclaredStanding, ): DeclaredOutcome[] { return outcomes.filter( (o): o is DeclaredOutcome => @@ -620,14 +626,53 @@ function withStanding( ); } +/** + * Every bucket the sweep reports, in ONE pass and ONE definition. + * + * Both the markdown summary and the stderr conclusion partition the same + * outcomes the same way, and they used to do it with two identical-looking + * blocks in different functions -- the drift hazard `effectivelyFailed`'s own + * doc warns about, one level up. `flakes` is here for the same reason: it was + * spelled out by hand in three places, twice differently. + */ +interface StandingPartition { + unexpectedFails: ScenarioOutcome[]; + expectedFails: DeclaredOutcome[]; + declaredButErrored: DeclaredOutcome[]; + unexpectedPasses: DeclaredOutcome[]; + /** Failed in a lane, did not fail isolated. Not a failure; reported so a + * red-looking run explains itself. */ + flakes: ScenarioOutcome[]; +} + +function partitionByStanding( + outcomes: readonly ScenarioOutcome[], +): StandingPartition { + return { + unexpectedFails: outcomes.filter( + (o) => standingOfOutcome(o) === "unexpected-fail", + ), + expectedFails: withStanding(outcomes, "expected-fail"), + declaredButErrored: withStanding(outcomes, "declared-but-errored"), + unexpectedPasses: withStanding(outcomes, "unexpected-pass"), + flakes: outcomes.filter( + (o) => + isFailure(o.verdict) && + !effectivelyFailed(o.verdict, o.isolatedRetry?.verdict), + ), + }; +} + /** * The driver's declaration, as the one sentence every report renders -- * `undefined` when it declared nothing about this scenario. * - * Written once because it was written three times: the markdown summary, the - * sweep's stderr list and the single-scenario line each rebuilt it, and the - * fourth copy had already drifted to a different separator. It is the same - * mistake unexpectedPassDetail() was extracted to fix, one level up. + * Written once because it was written three times, and one copy had already + * drifted to a different separator. The two sweep renderers -- the markdown + * verdict cell and the stderr list -- go through here. exitForSingleRun() + * deliberately does NOT: its sentences have to explain an exit code and + * interleave the declaration differently, so forcing them through one renderer + * would cost more than the labels it shares. * * Narrowing `expected` here also removes the `?.` chains the call sites needed: * they could not see that a standing of `expected-fail` implies an entry, so a @@ -637,7 +682,10 @@ function declarationNote(outcome: ScenarioOutcome): string | undefined { const { expected, verdict } = outcome; const retryVerdict = outcome.isolatedRetry?.verdict; if (!expected) return undefined; - switch (standingOf(verdict, expected, retryVerdict)) { + // standingOfOutcome(), not standingOf() on unpacked fields: unpacking here + // would be a third place that maps an outcome to a standing, and the exit + // code reads the other one. + switch (standingOfOutcome(outcome)) { case "expected-fail": return `EXPECTED FAIL: ${expected.reason} (${expected.finding})`; case "unexpected-pass": @@ -645,14 +693,16 @@ function declarationNote(outcome: ScenarioOutcome): string | undefined { "UNEXPECTED PASS: declared expected-failing, but " + unexpectedPassDetail(verdict, retryVerdict) ); - case "unexpected-fail": + case "declared-but-errored": return ( `DECLARED, BUT ERRORED (${expected.reason} -- ${expected.finding}): ` + declaredButErroredDetail(verdict, retryVerdict) ); case "ok": - // Unreachable: `expected` is non-undefined here, and standingOf() only - // answers "ok" for an undeclared scenario. + case "unexpected-fail": + // Unreachable: `expected` is non-undefined here, and both of these are + // the undeclared standings. There is no note to write for a scenario + // this driver said nothing about. return undefined; } } @@ -828,6 +878,7 @@ function hostLoad(): string { async function writeSummary( groupName: string, outcomes: ScenarioOutcome[], + parts: StandingPartition, ): Promise { const anyRetried = outcomes.some((o) => o.isolatedRetry !== undefined); @@ -870,8 +921,7 @@ async function writeSummary( `${naCount} NOT APPLICABLE (out of scope for this CSMS). Neither fails the sweep.`, ); } - const expectedFails = withStanding(outcomes, "expected-fail"); - const unexpectedPasses = withStanding(outcomes, "unexpected-pass"); + const { expectedFails, unexpectedPasses, declaredButErrored } = parts; if (expectedFails.length > 0) { notes.push( "", @@ -884,7 +934,6 @@ async function writeSummary( ), ); } - const declaredButErrored = withStanding(outcomes, "unexpected-fail"); if (declaredButErrored.length > 0) { notes.push( "", @@ -913,9 +962,7 @@ async function writeSummary( ); } if (anyRetried) { - const flakeCount = outcomes.filter( - (o) => o.isolatedRetry !== undefined && !isFailure(o.isolatedRetry.verdict), - ).length; + const flakeCount = parts.flakes.length; const confirmedCount = outcomes.filter( (o) => o.isolatedRetry !== undefined && isFailure(o.isolatedRetry.verdict), ).length; @@ -1026,29 +1073,21 @@ async function runGroupSweep( ); } - const summaryPath = await writeSummary(groupName, outcomes); + const parts = partitionByStanding(outcomes); + const summaryPath = await writeSummary(groupName, outcomes, parts); process.stderr.write(`[runner] results table: ${summaryPath}\n`); // A parallel-lane FAIL/ERROR that did not fail on its isolated retry is a // flake, not a real failure -- it does not fail the sweep. Everything below // reads that through effectivelyFailed(), via standingOf(). - const badOutcomes = outcomes.filter( - (o) => standingOfOutcome(o) === "unexpected-fail", - ); - const expectedFails = withStanding(outcomes, "expected-fail"); - const unexpectedPasses = withStanding(outcomes, "unexpected-pass"); + const { unexpectedFails, expectedFails, unexpectedPasses, declaredButErrored, flakes } = + parts; // A subset of badOutcomes, reported on its own line because it reads as a // regression of the DECLARATION when it is nothing of the sort: the entry is // almost certainly still good and something new stopped the scenario ever // reaching the CSMS. Without this the maintainer sees a red build on the one // row they were told would be red, and goes looking in the wrong place. - const declaredButErrored = withStanding(outcomes, "unexpected-fail"); - // A flake IS "failed, but not effectively" -- said through the shared - // predicate rather than spelled out again, which is what the comment above - // claims and what the previous hand-written copy quietly contradicted. - const flakeCount = outcomes.filter( - (o) => isFailure(o.verdict) && !effectivelyFailed(o.verdict, o.isolatedRetry?.verdict), - ).length; + const flakeCount = flakes.length; if (flakeCount > 0) { process.stderr.write( `[runner] ${flakeCount} parallel-only flake(s) in group '${groupName}' (did not fail on isolated retry) -- see ${summaryPath}\n`, @@ -1082,12 +1121,14 @@ async function runGroupSweep( ); } } - if (badOutcomes.length > 0) { + if (unexpectedFails.length > 0) { process.stderr.write( - `[runner] ${badOutcomes.length}/${outcomes.length} scenario(s) in group '${groupName}' FAILed or errored -- see ${summaryPath}\n`, + `[runner] ${unexpectedFails.length}/${outcomes.length} scenario(s) in group '${groupName}' FAILed or errored -- see ${summaryPath}\n`, ); } - if (badOutcomes.length > 0 || unexpectedPasses.length > 0) return 1; + // The disjunction lives in standing.ts, so a sixth standing cannot be added + // and silently left out of the exit code here. + if (outcomes.some((o) => endsTheBuild(standingOfOutcome(o)))) return 1; const partialCount = outcomes.filter((o) => o.verdict === "PARTIAL").length; const naCount = outcomes.filter((o) => o.verdict === "NOT APPLICABLE").length; @@ -1875,21 +1916,22 @@ function exitForSingleRun( `and ${unexpectedPassDetail(verdict)}.\n`, ); return 1; - case "unexpected-fail": + case "declared-but-errored": // Not reachable today -- a throw never reaches this function, so // `verdict` is never ERROR here and a declared FAIL is expected-fail. - // Written out anyway rather than folded into a default: "declared and - // errored" is the case the sweep exists to report, and the day this - // path learns to catch a throw it must not silently exit 1 with no word - // about the declaration it was carrying. + // Written out anyway rather than folded into a default: this is the case + // the sweep exists to report, and the day this path learns to catch a + // throw it must not silently exit 1 with no word about the declaration + // it was carrying. process.stderr.write( `[runner] ${templateId} DECLARED, BUT ERRORED (${expected.reason} -- ` + `${expected.finding}): ${declaredButErroredDetail(verdict)}.\n`, ); return 1; case "ok": - // Unreachable: standingOf() answers "ok" only for an undeclared - // scenario, and that returned above. - return 0; + case "unexpected-fail": + // Unreachable: both are the undeclared standings, and undeclared + // returned above. + return endsTheBuild(standing) ? 1 : 0; } } diff --git a/tck/standing.ts b/tck/standing.ts index 6c8e71d..9a61608 100644 --- a/tck/standing.ts +++ b/tck/standing.ts @@ -60,8 +60,7 @@ export function effectivelyFailed( verdict: Verdict, retryVerdict?: Verdict, ): boolean { - if (!isFailure(verdict)) return false; - return retryVerdict === undefined || isFailure(retryVerdict); + return isFailure(verdict) && isFailure(decisiveVerdict(verdict, retryVerdict)); } /** @@ -78,13 +77,25 @@ export function effectivelyFailed( * retry -- there is no "expected flaky", so an entry that passes any way at * all is an entry to look at. * - * `unexpected-fail` covers BOTH an undeclared failure and a declared scenario - * that ERRORed. See {@link standingOf} for why the second belongs here. + * `declared-but-errored` is the fifth, and it earns a member of its own rather + * than hiding inside `unexpected-fail`: README.md and CONTRIBUTING.md both name + * it, the sweep reports it on its own line, and the reaction it calls for is + * the opposite one -- keep the entry, chase the crash. Expressed as + * "`unexpected-fail` AND a declaration is present" it was a conjunct living in + * a filter, so two call sites asking the same-looking question ten lines apart + * quietly got different sets. */ -export type SweepStanding = - | "ok" +export type SweepStanding = "ok" | "unexpected-fail" | DeclaredStanding; + +/** + * The standings that imply the driver declared this scenario. Split out so the + * implication is a type rather than a sentence: a helper that filters outcomes + * by one of these can promise the declaration is there, and the compiler + * checks the promise instead of a reader checking a comment. + */ +export type DeclaredStanding = | "expected-fail" - | "unexpected-fail" + | "declared-but-errored" | "unexpected-pass"; /** @@ -110,7 +121,7 @@ export type SweepStanding = * | no | no | any | ok | * | no | yes | any | unexpected-fail | * | yes | yes | FAIL | expected-fail | - * | yes | yes | ERROR | unexpected-fail | + * | yes | yes | ERROR | declared-but-errored | * | yes | no | any | unexpected-pass | */ export function standingOf( @@ -125,7 +136,7 @@ export function standingOf( if (failed) { return decisiveVerdict(verdict, retryVerdict) === "FAIL" ? "expected-fail" - : "unexpected-fail"; + : "declared-but-errored"; } // Everything else is an unexpected pass, INCLUDING NOT APPLICABLE. That one // is not merely theoretical: check-driver rejects an id declared both ways, @@ -137,42 +148,73 @@ export function standingOf( return "unexpected-pass"; } +/** The standings that make the process exit non-zero. Exported so no caller + * has to rebuild the disjunction and forget a member -- which is exactly what + * happened while `declared-but-errored` was hiding inside `unexpected-fail`. */ +export function endsTheBuild(standing: SweepStanding): boolean { + return standing !== "ok" && standing !== "expected-fail"; +} + +/** + * WHICH KIND of unexpected pass, as a value rather than as prose. + * + * `standingOf` collapses four genuinely different situations onto one + * `unexpected-pass`, and only ONE of them is evidence that the CSMS was fixed. + * Since the action the report implies is DELETING A RECORDED FINDING, that + * distinction is the load-bearing part -- so it is computed here, from the + * same two inputs, and {@link unexpectedPassDetail} is a rendering of it. + * + * Splitting it out is what lets a guard assert the distinction as a table. + * Asserting the SENTENCES instead was tried and does not work: a guard that + * greps for "looks fixed" passes happily when the degraded case is reworded to + * say "looks resolved", which is the very defect it was meant to catch. + */ +export type UnexpectedPassKind = "fixed" | "degraded" | "never-ran" | "flaky"; + +export function unexpectedPassKind( + verdict: Verdict, + retryVerdict?: Verdict, +): UnexpectedPassKind { + if (verdict === "NOT APPLICABLE") return "never-ran"; + if (verdict === "PARTIAL") return "degraded"; + return retryVerdict !== undefined && isFailure(verdict) ? "flaky" : "fixed"; +} + /** * Why an unexpected pass is one, in the words a reader needs to tell a fix * from a flake from a contradiction from an unmeasured run -- without reading * this file. * - * The wording carries real weight, because the action it implies is DELETING A - * RECORDED FINDING. Only one of the cases below is actually evidence that the - * CSMS was fixed; saying so for the others would talk a maintainer into - * throwing away a finding that still holds. + * A rendering of {@link unexpectedPassKind}, and nothing more: the decision is + * there, where it can be checked; the prose is here, where it cannot. */ export function unexpectedPassDetail( verdict: Verdict, retryVerdict?: Verdict, ): string { - if (verdict === "NOT APPLICABLE") { - return ( - "it never ran (NOT APPLICABLE) -- the scope table and the " + - "expected-failure list contradict each other, so the contradiction is " + - "what to fix, and deleting the entry may be the wrong half" - ); - } - if (verdict === "PARTIAL") { - return ( - "it came back PARTIAL -- at least one check was SKIPPED because the " + - "driver could not evaluate it, so this is NOT evidence that the finding " + - "is fixed; find out which check degraded before deleting the entry" - ); - } - if (retryVerdict !== undefined && isFailure(verdict)) { - return ( - `it failed in the parallel lane and ${retryVerdict} on its isolated ` + - "retry -- either upstream fixed it, or the scenario is flaky and the " + - "flake is the bug to fix" - ); + switch (unexpectedPassKind(verdict, retryVerdict)) { + case "never-ran": + return ( + "it never ran (NOT APPLICABLE) -- the scope table and the " + + "expected-failure list contradict each other, so the contradiction is " + + "what to fix, and deleting the entry may be the wrong half" + ); + case "degraded": + return ( + "it came back PARTIAL -- at least one check was SKIPPED because the " + + "driver could not evaluate it, so this is NOT evidence that the " + + "finding is fixed; find out which check degraded before deleting the " + + "entry" + ); + case "flaky": + return ( + `it failed in the parallel lane and ${retryVerdict} on its isolated ` + + "retry -- either upstream fixed it, or the scenario is flaky and the " + + "flake is the bug to fix" + ); + case "fixed": + return `it came back ${verdict} outright -- the finding looks fixed`; } - return `it came back ${verdict} outright -- the finding looks fixed`; } /** diff --git a/tests/expected-failure-standing.ts b/tests/expected-failure-standing.ts index 1321a2a..8944835 100644 --- a/tests/expected-failure-standing.ts +++ b/tests/expected-failure-standing.ts @@ -28,9 +28,12 @@ import type { ExpectedFailureEntry } from "../tck/expected"; import { declaredButErroredDetail, effectivelyFailed, + endsTheBuild, standingOf, unexpectedPassDetail, + unexpectedPassKind, type SweepStanding, + type UnexpectedPassKind, type Verdict, } from "../tck/standing"; @@ -65,9 +68,9 @@ const TABLE: Row[] = [ // --- declared, and it ERRORED: a declaration excuses an answer, not a // crash. These four rows are the finding this guard was written for. - { verdict: "ERROR", declared: true, expect: "unexpected-fail", because: "an ERROR never produced the answer the entry describes" }, - { verdict: "ERROR", retry: "ERROR", declared: true, expect: "unexpected-fail", because: "errored isolated too -- still not the declared failure" }, - { verdict: "FAIL", retry: "ERROR", declared: true, expect: "unexpected-fail", because: "the isolated retry is the arbiter, and it crashed" }, + { verdict: "ERROR", declared: true, expect: "declared-but-errored", because: "an ERROR never produced the answer the entry describes" }, + { verdict: "ERROR", retry: "ERROR", declared: true, expect: "declared-but-errored", because: "errored isolated too -- still not the declared failure" }, + { verdict: "FAIL", retry: "ERROR", declared: true, expect: "declared-but-errored", because: "the isolated retry is the arbiter, and it crashed" }, { verdict: "ERROR", retry: "FAIL", declared: true, expect: "expected-fail", because: "the isolated retry is the arbiter: the parallel ERROR was the lane, and isolated it reproduced the declared failure" }, // --- declared, and it did NOT fail: the half that lets the list shrink -- @@ -105,39 +108,68 @@ check( "a non-failure verdict is now treated as a failure.", ); -// The message an UNEXPECTED PASS carries decides whether a maintainer deletes -// a still-valid finding, so the two cases that are NOT evidence of a fix must -// not claim to be. -check( - !unexpectedPassDetail("PARTIAL").includes("looks fixed"), - "a PARTIAL now reads as evidence the finding is fixed; it is evidence of " + - "nothing, because the check that caught it may be the one that SKIPPED.", -); -check( - !unexpectedPassDetail("NOT APPLICABLE").includes("looks fixed"), - "a NOT APPLICABLE now reads as evidence the finding is fixed; it means the " + - "scenario never ran.", -); -check( - unexpectedPassDetail("PASS").includes("looks fixed"), - "an outright PASS no longer reads as evidence the finding is fixed, which " + - "is the one case where it is.", -); +// Which STANDINGS end the build, asserted against the union rather than +// against a disjunction retyped at the call site. A sixth standing that +// nobody classifies shows up here. +const ALL_STANDINGS: SweepStanding[] = [ + "ok", + "expected-fail", + "unexpected-fail", + "declared-but-errored", + "unexpected-pass", +]; +const SHOULD_END_BUILD = new Set([ + "unexpected-fail", + "declared-but-errored", + "unexpected-pass", +]); +for (const standing of ALL_STANDINGS) { + check( + endsTheBuild(standing) === SHOULD_END_BUILD.has(standing), + `endsTheBuild("${standing}") is now ${endsTheBuild(standing)} -- only an ` + + "unexcused failure, a declared crash and a declared pass end the build.", + ); +} + +// WHICH KIND of unexpected pass, as a value table. +// +// This replaces three assertions that grepped the MESSAGES for "looks fixed". +// They were both brittle and blind, and the blindness was demonstrated rather +// than suspected: rewording the PARTIAL branch to "the finding looks resolved" +// -- the maintainer-misleading defect the assertions existed to catch -- +// left them green, because the token they matched had moved. The kind is what +// the rule actually computes, so it is what can be pinned. +const KINDS: { verdict: Verdict; retry?: Verdict; kind: UnexpectedPassKind; because: string }[] = [ + { verdict: "PASS", kind: "fixed", because: "the one case that IS evidence the CSMS was fixed" }, + { verdict: "PARTIAL", kind: "degraded", because: "a check was SKIPPED, so nothing was measured either way" }, + { verdict: "NOT APPLICABLE", kind: "never-ran", because: "the scope table and the list contradict each other" }, + { verdict: "FAIL", retry: "PASS", kind: "flaky", because: "failed in a lane, passed isolated" }, + { verdict: "ERROR", retry: "PASS", kind: "flaky", because: "same, from the ERROR side" }, +]; +for (const row of KINDS) { + const got = unexpectedPassKind(row.verdict, row.retry); + check( + got === row.kind, + `unexpectedPassKind(${row.verdict}${row.retry ? ` -> ${row.retry}` : ""}) ` + + `is "${got}", expected "${row.kind}" -- ${row.because}.`, + ); +} -// The mirror of the three above. This message points the reader AWAY from the -// entry -- the declaration is probably intact and the crash is new -- so -// inverting it would send them to delete a finding that still holds, which is -// the same damage from the opposite direction. +// The four messages must stay tellable apart. This is the one PROSE property +// worth pinning: it does not name a phrase, so any reword survives it, while a +// copy-paste that makes two kinds read alike does not. Beyond this, whether a +// sentence MEANS what it should is a review concern -- no string match decides +// it, and pretending otherwise is what the deleted assertions did. +const MESSAGES = KINDS.map((row) => unexpectedPassDetail(row.verdict, row.retry)); check( - declaredButErroredDetail("ERROR").includes("still good"), - "a declared scenario that errored no longer reads as 'the entry is " + - "probably still good', so the message now points at the declaration " + - "rather than at the crash.", + new Set(MESSAGES).size === new Set(KINDS.map((r) => r.kind)).size, + "two unexpected-pass kinds now render the same sentence, so a reader cannot " + + "tell which evidence they are looking at.", ); check( - declaredButErroredDetail("FAIL", "ERROR").includes("isolated retry"), - "the FAIL-then-ERROR case no longer says where the ERROR came from, so a " + - "reader cannot tell a crashing retry from a crashing lane.", + unexpectedPassDetail("PASS") !== declaredButErroredDetail("ERROR"), + "an unexpected pass and a declared crash now read alike, and they call for " + + "opposite actions -- delete the entry versus keep it and chase the crash.", ); function check(condition: boolean, failure: string): void { diff --git a/tests/spec-invariants.sh b/tests/spec-invariants.sh index 6bfebcf..15fb04c 100755 --- a/tests/spec-invariants.sh +++ b/tests/spec-invariants.sh @@ -67,14 +67,21 @@ trap 'rm -rf "$work"' EXIT INT TERM # generated artifact is ever written back into the repo by this test. # # node_modules is EXCLUDED FROM THE COPY rather than deleted after it, and that -# is a bug fix, not a speed-up. `cp -r /src/. /work/` followed by +# is a bug fix rather than a speed-up. `cp -r /src/. /work/` followed by # `rm -rf node_modules` copied ~640 files across the read-only bind mount for # the sole purpose of deleting them, and on that mount the copy failed # intermittently -- roughly one run in three on macOS -- with busybox cp # reporting "can't create link" for paths whose parent directory it had not # created yet. Measured on origin/main, so it predates any caller: clean # checkout 6/6 green, same checkout after `bun install` 5/6. Never copying the -# tree is what removes the race; the guard's own inputs are 350 files. +# files is what removes the race; the guard's own inputs are ~120. +# +# .git goes for the same reason, and it is the larger half in a normal clone: +# neither extractor reads it and `bun add` does not either, yet at 1 600 files +# it is several times the cost of everything the copy legitimately carries -- +# and every one of those files is another draw in the lottery above. Free here, +# where this workspace is a worktree and .git is a one-line pointer; not free +# for anyone running the gate in a full clone. # # find, not a glob: dotfiles have to come across (tsconfig lives beside # .gitignore, and a missing one changes what the extractors see), and @@ -85,7 +92,7 @@ if ! docker run --rm \ "$bun_image" \ sh -c "set -e mkdir -p /work - find /src -mindepth 1 -maxdepth 1 ! -name node_modules -exec cp -r {} /work/ ';' + find /src -mindepth 1 -maxdepth 1 ! -name node_modules ! -name .git -exec cp -r {} /work/ ';' cd /work bun add -d 'typescript@$typescript_version' >/dev/null 2>&1 bun tools/extract-assert-inventory.ts tck/specs > /out/ASSERT-INVENTORY.txt diff --git a/types/tck/standing.d.ts b/types/tck/standing.d.ts index 7855873..903a894 100644 --- a/types/tck/standing.d.ts +++ b/types/tck/standing.d.ts @@ -50,10 +50,22 @@ export declare function effectivelyFailed(verdict: Verdict, retryVerdict?: Verdi * retry -- there is no "expected flaky", so an entry that passes any way at * all is an entry to look at. * - * `unexpected-fail` covers BOTH an undeclared failure and a declared scenario - * that ERRORed. See {@link standingOf} for why the second belongs here. + * `declared-but-errored` is the fifth, and it earns a member of its own rather + * than hiding inside `unexpected-fail`: README.md and CONTRIBUTING.md both name + * it, the sweep reports it on its own line, and the reaction it calls for is + * the opposite one -- keep the entry, chase the crash. Expressed as + * "`unexpected-fail` AND a declaration is present" it was a conjunct living in + * a filter, so two call sites asking the same-looking question ten lines apart + * quietly got different sets. */ -export type SweepStanding = "ok" | "expected-fail" | "unexpected-fail" | "unexpected-pass"; +export type SweepStanding = "ok" | "unexpected-fail" | DeclaredStanding; +/** + * The standings that imply the driver declared this scenario. Split out so the + * implication is a type rather than a sentence: a helper that filters outcomes + * by one of these can promise the declaration is there, and the compiler + * checks the promise instead of a reader checking a comment. + */ +export type DeclaredStanding = "expected-fail" | "declared-but-errored" | "unexpected-pass"; /** * A DECLARATION EXCUSES AN ANSWER, NEVER A CRASH. * @@ -77,19 +89,37 @@ export type SweepStanding = "ok" | "expected-fail" | "unexpected-fail" | "unexpe * | no | no | any | ok | * | no | yes | any | unexpected-fail | * | yes | yes | FAIL | expected-fail | - * | yes | yes | ERROR | unexpected-fail | + * | yes | yes | ERROR | declared-but-errored | * | yes | no | any | unexpected-pass | */ export declare function standingOf(verdict: Verdict, expected: ExpectedFailureEntry | undefined, retryVerdict?: Verdict): SweepStanding; +/** The standings that make the process exit non-zero. Exported so no caller + * has to rebuild the disjunction and forget a member -- which is exactly what + * happened while `declared-but-errored` was hiding inside `unexpected-fail`. */ +export declare function endsTheBuild(standing: SweepStanding): boolean; +/** + * WHICH KIND of unexpected pass, as a value rather than as prose. + * + * `standingOf` collapses four genuinely different situations onto one + * `unexpected-pass`, and only ONE of them is evidence that the CSMS was fixed. + * Since the action the report implies is DELETING A RECORDED FINDING, that + * distinction is the load-bearing part -- so it is computed here, from the + * same two inputs, and {@link unexpectedPassDetail} is a rendering of it. + * + * Splitting it out is what lets a guard assert the distinction as a table. + * Asserting the SENTENCES instead was tried and does not work: a guard that + * greps for "looks fixed" passes happily when the degraded case is reworded to + * say "looks resolved", which is the very defect it was meant to catch. + */ +export type UnexpectedPassKind = "fixed" | "degraded" | "never-ran" | "flaky"; +export declare function unexpectedPassKind(verdict: Verdict, retryVerdict?: Verdict): UnexpectedPassKind; /** * Why an unexpected pass is one, in the words a reader needs to tell a fix * from a flake from a contradiction from an unmeasured run -- without reading * this file. * - * The wording carries real weight, because the action it implies is DELETING A - * RECORDED FINDING. Only one of the cases below is actually evidence that the - * CSMS was fixed; saying so for the others would talk a maintainer into - * throwing away a finding that still holds. + * A rendering of {@link unexpectedPassKind}, and nothing more: the decision is + * there, where it can be checked; the prose is here, where it cannot. */ export declare function unexpectedPassDetail(verdict: Verdict, retryVerdict?: Verdict): string; /** From 695c1d73a0bb646e365fb302082782f684836025 Mon Sep 17 00:00:00 2001 From: Julien Herr Date: Fri, 14 Aug 2026 14:42:49 +0200 Subject: [PATCH 11/11] docs: state the ERROR rule once, and where it belongs Review follow-ups, all documentation or comment placement. No behaviour change: the tck/main.ts diff is comments only. README.md said "FAIL and ERROR exit non-zero -- unless your driver said so first", which stopped being true when a declaration lost the power to excuse an ERROR. It now says which of the two a declaration excuses, matching the Expected failures section thirty lines below that already got it right. drivers/citrineos/README.md carried the Blocked finding as two gap-table rows, the second deferring to the first for the mechanism and adding the observed result. One row now, mechanism and evidence together. The duplication predates this branch, but both rows were edited here. The comment about reporting declared-but-errored on its own line had been left above `const flakeCount`, which it does not describe -- it drifted there when the buckets moved into partitionByStanding(). Moved to the block it explains, and its "a subset of badOutcomes" opening corrected: since DECLARED, BUT ERRORED became a standing of its own, it is disjoint from unexpected-fail rather than contained in it. Also fixes a sentence in the CitrineOS scope row that read "sends the Authorize the scenario asks for". --- README.md | 4 +++- VENDOR.md | 2 +- drivers/citrineos/README.md | 3 +-- drivers/citrineos/scope.ts | 3 ++- patches/tck/main.ts.patch | 35 ++++++++++++++++++----------------- tck/main.ts | 17 +++++++++-------- 6 files changed, 34 insertions(+), 30 deletions(-) diff --git a/README.md b/README.md index b60e2e8..9f80716 100644 --- a/README.md +++ b/README.md @@ -174,7 +174,9 @@ the check could not be evaluated" is a real outcome: because a driver answered with `unverifiable("")` instead of inventing a value. Exit code 0. -`FAIL` and `ERROR` exit non-zero — unless your driver said so first. +`FAIL` and `ERROR` exit non-zero. A driver may declare a scenario +expected-failing, which excuses its `FAIL` — never its `ERROR`, and never a +declared scenario that stops failing. See below. ## Expected failures diff --git a/VENDOR.md b/VENDOR.md index 0cca3aa..8012336 100644 --- a/VENDOR.md +++ b/VENDOR.md @@ -57,7 +57,7 @@ strongest available demonstration that the core names no CSMS. | `tck/specs/index.ts` | `upstream-verbatim` | `scripts/steve-verify/runner/specs/index.ts` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `be8595765f4d66965bfd58498622c26a696962fabae8a2700f080ae5cd55d832` | `—` | | `tck/assert.ts` | `upstream-patched` | `src/cp/application/verification/assert.ts` | `2431f5f6c0df997d4d821d9af55689c1f0f2df199de1e9e4ed6f3fbaad4fc89e` | `76d0f293db4f6ecc5768affea0ca76d2841c147fa3687dbea1f9e950bfbd9298` | `patches/tck/assert.ts.patch` | | `tck/sim.ts` | `upstream-patched` | `scripts/steve-verify/runner/sim.ts` | `2bf2f78afe3434e7139cd62c3ff6d70f02defd39dd700611e7c5f7614260cd35` | `45f897f273ea73227bb9b30766127eb9ecae0a319ab9048cbc30428fafff7f32` | `patches/tck/sim.ts.patch` | -| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `aa1153bc54894707510654d818382b1169406a2d2927822ac018c2e470fa21a1` | `patches/tck/main.ts.patch` | +| `tck/main.ts` | `upstream-patched` | `scripts/steve-verify/runner/main.ts` | `a757b0d35d29c7627336c0e858ad7d2f305a33c0acad819b5296f3847382f4e2` | `6b6c2f6381b418bc7df4d03126d07a64419b89440e4faf8ca4e7c1e924b32453` | `patches/tck/main.ts.patch` | | `tck/driver.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/index.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | | `tck/driver-registry.ts` | `local-upstreamable` | `—` | `—` | `—` | `—` | diff --git a/drivers/citrineos/README.md b/drivers/citrineos/README.md index 0b87513..9c33d9a 100644 --- a/drivers/citrineos/README.md +++ b/drivers/citrineos/README.md @@ -275,10 +275,9 @@ a `reason` that cannot name the limitation is `CONDITIONAL`, not | **No OCPP 1.6 reservation endpoints.** No `@AsMessageEndpoint` binds `ReserveNow` or `CancelReservation` to `OCPPVersion.OCPP1_6`. The 1.6 schemas exist, the `Reservations` table exists, `evdriver.responses` lists both actions — nothing routes them, and no 1.6 response handler exists either. | 7 scenarios `NOT APPLICABLE`; the driver omits `records.reservations` entirely. | Verified at `v1.9.1`, `v2.0.0-beta1` and `main`. | | **Local auth list is v2-only.** `EVDriverOcpp16Api` gained `sendLocalList` / `getLocalListVersion` in the v2 line. | 6 scenarios, drivable only on the pinned prerelease. | `packages/core/src/modules/EVDriver/src/module/1.6/MessageApi.ts` | | **No charging-profile registry.** `ChargingProfiles` has no `description` or `name` column, and nothing to look one up by. | `refByDescription` resolves from this driver's own catalogue instead. Not a scenario cost: OCPP 1.6 carries the profile inline. | `packages/core/src/dal/layers/drizzle/schema/ChargingProfile.ts`, and [`profiles.ts`](profiles.ts) | -| **`Blocked` is unreachable from the 1.6 `Authorize` path.** The handler reaches its status mapper only through the `status === Accepted` branch, so a stored `Blocked` falls through to the default `Invalid`. The only route to a real `Blocked` is an `IAuthorizer`, and the container registers `authorizers: asValue([])` with no setting that changes it. | TC_023.3 — `scope.ts` keeps the row DRIVABLE, `expected.ts` declares the red. | `AuthorizeRequestOcpp16Handler.ts`, `apps/ocpp-server/src/container.ts` | +| **`Blocked` is unreachable from the 1.6 `Authorize` path.** The handler reaches its status mapper only through the `status === Accepted` branch, so a stored `Blocked` falls through to the default `Invalid`. The only route to a real `Blocked` is an `IAuthorizer`, and the container registers `authorizers: asValue([])` with no setting that changes it. | **TC_023.3 fails**, deterministically: CitrineOS answers `{"idTagInfo":{"status":"Invalid"}}` where the scenario requires `Blocked`. Observed 3 runs out of 3. `scope.ts` keeps the row DRIVABLE and `expected.ts` declares the red, so the sweep reports it as `EXPECTED FAIL` and the job stays blocking — and `UNEXPECTED PASS` the day it is fixed. | `AuthorizeRequestOcpp16Handler.ts`, `apps/ocpp-server/src/container.ts` | | **No REST for `Authorizations`.** `EVDriverDataApi` exposes exactly one route, a read-only local-list-version GET. | `driver provision` writes fixtures through GraphQL. | [`provision.ts`](provision.ts) | | **Four foreign keys reference `Authorizations`, none cascading**: `Transactions.authorizationId`, `LocalListAuthorizations.authorizationId`, `LocalListAuthorizations.groupAuthorizationId`, and the self-reference `Authorizations.groupAuthorizationId`. | `teardown` derives its guards from the foreign keys Hasura reports instead of listing them, so a fifth on a future CitrineOS is picked up rather than aborting the whole delete. Guarding only the first was measured to leave *every* fixture in place, because psql ran the script in one implicit transaction. | Read from the foreign keys Hasura derives; see `references()` in [`provision.ts`](provision.ts). | -| **`Blocked` is unreachable from the 1.6 `Authorize` path** *(see the row above for the mechanism)*. | **TC_023.3 fails**, deterministically: CitrineOS answers `{"idTagInfo":{"status":"Invalid"}}` where the scenario requires `Blocked`. Observed 3 runs out of 3. | Kept red on purpose, and declared: `expected.ts` makes it an `EXPECTED FAIL` so the job stays blocking, and an `UNEXPECTED PASS` the day it is fixed. | | **No 1.6 request handler for `FirmwareStatusNotification`.** Every one the charge point sends is answered with `[4,…,"NotSupported","No handler found for action: FirmwareStatusNotification at module configuration"]` — 9 across the three TC_044 logs, and the only CALLERROR the CSMS emits anywhere in the suite. | **A non-conformance, and one this suite does not detect.** OCA `TC_044_{1,2,3}_CSMS` put steps 4 and 6 on the Central System — *"The Central responds with a FirmwareStatusNotification.conf"* — and a CALLERROR is not that conf. Our scenarios pass because they assert only on the statuses the charge point *sent*. See below. | `packages/core/src/handlers/requests/1.6/` — `DiagnosticsStatusNotification` has one, `FirmwareStatusNotification` does not. No ticket upstream. | | **No 1.6 response handler for `UnlockConnector` or `UpdateFirmware`.** The Calls are routed and sent; the CallResults are answered with the same `NotSupported` CALLERROR. | Harmless — the six affected scenarios all pass. | `packages/core/src/handlers/responses/1.6/` | | **`GetConfiguration` is batched server-side.** The endpoint splits a request into batches of the station's stored `GetConfigurationMaxKeys`. | *Not* a problem in practice: an unprovisioned station has no such value, so the request stays one `GetConfiguration` on the wire and both TC_019 scenarios pass. Listed because provisioning that key would change it. | `Configuration/src/module/1.6/MessageApi.ts` | diff --git a/drivers/citrineos/scope.ts b/drivers/citrineos/scope.ts index f06446d..819baa4 100644 --- a/drivers/citrineos/scope.ts +++ b/drivers/citrineos/scope.ts @@ -102,7 +102,8 @@ const V2_SCOPE = { // cannot drift. "cert16-tc023-3-authorize-blocked": d( `${BLOCKED_UNREACHABLE} Expressible and driven: the driver sends the ` + - "Authorize the scenario asks for and reads the answer back. What comes " + + "Authorize request the scenario asks for and reads the answer back. " + + "What comes " + "back is Invalid where OCPP 1.6 requires Blocked, which is a finding " + "against CitrineOS rather than a gap in this driver -- declared in " + "expected.ts.", diff --git a/patches/tck/main.ts.patch b/patches/tck/main.ts.patch index 6b10af7..3c3cdc9 100644 --- a/patches/tck/main.ts.patch +++ b/patches/tck/main.ts.patch @@ -1224,7 +1224,7 @@ "(a sequential sweep is already isolated).\n", ); } -@@ -588,45 +1062,86 @@ +@@ -588,45 +1062,87 @@ for (const o of outcomes) { const retrySuffix = o.isolatedRetry ? ` [isolated retry: ${o.isolatedRetry.verdict}${ @@ -1256,16 +1256,11 @@ - const flakeCount = outcomes.filter( - (o) => o.verdict !== "PASS" && o.isolatedRetry?.verdict === "PASS", - ).length; -+ // A parallel-lane FAIL/ERROR that did not fail on its isolated retry is a -+ // flake, not a real failure -- it does not fail the sweep. Everything below -+ // reads that through effectivelyFailed(), via standingOf(). + const { unexpectedFails, expectedFails, unexpectedPasses, declaredButErrored, flakes } = + parts; -+ // A subset of badOutcomes, reported on its own line because it reads as a -+ // regression of the DECLARATION when it is nothing of the sort: the entry is -+ // almost certainly still good and something new stopped the scenario ever -+ // reaching the CSMS. Without this the maintainer sees a red build on the one -+ // row they were told would be red, and goes looking in the wrong place. ++ // A parallel-lane FAIL/ERROR that did not fail on its isolated retry is a ++ // flake, not a real failure -- it does not fail the sweep. Everything here ++ // reads that through effectivelyFailed(), via standingOf(). + const flakeCount = flakes.length; if (flakeCount > 0) { process.stderr.write( @@ -1281,6 +1276,12 @@ ); - process.exit(1); } ++ // Its own standing, and its own line, because it reads as a regression of the ++ // DECLARATION when it is nothing of the sort: the entry is almost certainly ++ // still good and something new stopped the scenario ever reaching the CSMS. ++ // Folded in with the ordinary failures, the maintainer sees a red build on ++ // the one row they were told would be red, and goes looking in the wrong ++ // place. + if (declaredButErrored.length > 0) { + process.stderr.write( + `[runner] ${declaredButErrored.length} scenario(s) in group '${groupName}' are declared expected-failing and ERRORED, which is not the declared failure:\n`, @@ -1333,7 +1334,7 @@ } // --------------------------------------------------------------------------- -@@ -642,6 +1157,8 @@ +@@ -642,6 +1158,8 @@ cpId: string; connector?: number; timeoutSecs?: number; @@ -1342,7 +1343,7 @@ } function requireValue(argv: string[], index: number, flag: string): string { -@@ -663,37 +1180,80 @@ +@@ -663,37 +1181,80 @@ return n; } @@ -1440,7 +1441,7 @@ if (argv[0] === "run-all") { group = "all"; -@@ -708,18 +1268,27 @@ +@@ -708,18 +1269,27 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1471,7 +1472,7 @@ process.exit(1); } -@@ -749,9 +1318,11 @@ +@@ -749,9 +1319,11 @@ case "--retry-failed-isolated": retryFailedIsolated = true; break; @@ -1484,7 +1485,7 @@ process.exit(1); } } -@@ -764,35 +1335,495 @@ +@@ -764,35 +1336,495 @@ cpId, connector, timeoutSecs, @@ -1513,7 +1514,7 @@ + } + return [...seen].map(([templateId, group]) => ({ templateId, group })); +} - ++ +function listScenarios(argv: string[]): number { + let group: string | undefined; + let asJson = false; @@ -1696,7 +1697,7 @@ + ); + } + } -+ + + if (capabilities) { + const known = new Set(CSMS_OPERATION_ACTIONS); + const unknown = [...capabilities.operations].filter( @@ -1996,7 +1997,7 @@ } const spec = SPECS_BY_TEMPLATE_ID.get(args.templateId); -@@ -800,18 +1831,107 @@ +@@ -800,18 +1832,107 @@ process.stderr.write( `Unknown template id: ${args.templateId} (known: ${[...SPECS_BY_TEMPLATE_ID.keys()].join(", ")})\n`, ); diff --git a/tck/main.ts b/tck/main.ts index a428347..f7b9e99 100644 --- a/tck/main.ts +++ b/tck/main.ts @@ -1077,16 +1077,11 @@ async function runGroupSweep( const summaryPath = await writeSummary(groupName, outcomes, parts); process.stderr.write(`[runner] results table: ${summaryPath}\n`); - // A parallel-lane FAIL/ERROR that did not fail on its isolated retry is a - // flake, not a real failure -- it does not fail the sweep. Everything below - // reads that through effectivelyFailed(), via standingOf(). const { unexpectedFails, expectedFails, unexpectedPasses, declaredButErrored, flakes } = parts; - // A subset of badOutcomes, reported on its own line because it reads as a - // regression of the DECLARATION when it is nothing of the sort: the entry is - // almost certainly still good and something new stopped the scenario ever - // reaching the CSMS. Without this the maintainer sees a red build on the one - // row they were told would be red, and goes looking in the wrong place. + // A parallel-lane FAIL/ERROR that did not fail on its isolated retry is a + // flake, not a real failure -- it does not fail the sweep. Everything here + // reads that through effectivelyFailed(), via standingOf(). const flakeCount = flakes.length; if (flakeCount > 0) { process.stderr.write( @@ -1098,6 +1093,12 @@ async function runGroupSweep( `[runner] ${expectedFails.length} scenario(s) in group '${groupName}' failed as this driver declared they would -- not a build failure; see ${summaryPath}\n`, ); } + // Its own standing, and its own line, because it reads as a regression of the + // DECLARATION when it is nothing of the sort: the entry is almost certainly + // still good and something new stopped the scenario ever reaching the CSMS. + // Folded in with the ordinary failures, the maintainer sees a red build on + // the one row they were told would be red, and goes looking in the wrong + // place. if (declaredButErrored.length > 0) { process.stderr.write( `[runner] ${declaredButErrored.length} scenario(s) in group '${groupName}' are declared expected-failing and ERRORED, which is not the declared failure:\n`,