Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 103 additions & 9 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,14 @@ jobs:

# Offline: reads the driver module and the scenario registry, nothing
# else. No CSMS, no container, no credentials.
- name: Reference driver agrees with the core
run: bun run check:reference-driver
- name: SteVe driver agrees with the core
run: bun run check:steve-driver

# The same check for the second driver, and it earns its place: this is
# the one whose scope table has NOT_APPLICABLE rows, so it is the only
# place where adding or renaming a scenario silently stops being covered.
- name: CitrineOS driver agrees with the core
run: bun run check:citrineos-driver

- name: Core names no CSMS
run: bash tests/generic-core.sh
Expand All @@ -60,10 +66,15 @@ jobs:
- name: Shell guards lint clean
run: shellcheck tests/*.sh tools/*.sh

# Everything above is offline and proves the driver type-checks. This proves
# it works: a real SteVe, real containers, real OCPP-J frames. Without it the
# reference driver can rot green -- nothing else here would notice.
e2e:
# Everything above is offline and proves the drivers type-check. The two e2e
# jobs prove they work: real containers, real OCPP-J frames. They are peers,
# and each answers a question the other cannot -- see e2e-citrineos below.
#
# This one: SteVe is the CSMS the 47 scenarios were originally written
# against, so it is where a capability the harness LOST shows up first. Its
# scope table claims every scenario; a row that had to be demoted would mean
# the core dropped something. Without this job that can rot green unnoticed.
e2e-steve:
runs-on: ubuntu-latest
needs: check
# ~10 min of sweep plus image pull, a cold SteVe boot and the one-off
Expand All @@ -89,7 +100,7 @@ jobs:
# two pinned images and cold-booting SteVe (its healthcheck allows 150s
# for Flyway) shares nothing with installing bun, so the two may as well
# overlap. The blocking wait below is then usually already satisfied.
- name: Start the reference CSMS
- name: Start SteVe
run: docker compose -f drivers/steve/compose.yaml up -d

- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
Expand Down Expand Up @@ -119,8 +130,8 @@ jobs:
# barely moves with runner speed. Across 3 lanes, core is ~3 min and all
# 44 are ~10 min. Roughly 7 minutes buys back the 29 scenarios that core
# leaves out -- on precisely the branch where a regression is introduced,
# and for a job whose entire reason to exist is that the reference driver
# cannot rot green. Cheap coverage is not worth a stale signal.
# and for a job whose entire reason to exist is that this driver cannot
# rot green. Cheap coverage is not worth a stale signal.
#
# --retry-failed-isolated is not optional here: the runner warns that
# parallel lanes are not fully isolated, so a bare --parallel turns
Expand Down Expand Up @@ -148,3 +159,86 @@ jobs:
- name: Tear down
if: always()
run: docker compose -f drivers/steve/compose.yaml down -v

# The peer of the job above, answering the question it cannot: CitrineOS had
# no part in writing the scenarios, so this is where the core would be caught
# quietly re-acquiring a dependency on how SteVe happens to behave. A
# CSMS-neutrality claim that only one CSMS ever exercises is a claim, not a
# check.
#
# continue-on-error, and NOT because the job is unreliable: it is expected to
# be red. Two of the 47 scenarios fail against the pinned CitrineOS, and both
# are kept red deliberately rather than demoted in the scope table --
# tck/scope.ts forbids demoting a row to make a red scenario go away, and
# these are findings about CitrineOS:
#
# cert16-tc023-3-authorize-blocked deterministic (3 runs of 3): CitrineOS
# answers Invalid where OCPP 1.6 requires
# Blocked.
# cert16-tc044-2-firmware-download-failed
# flaky (1 pass in 3), thinnest timing
# margin of any scenario.
#
# So this job's value is the OTHER 45, and the artifact it uploads. Turn it
# blocking only once both rows are green -- i.e. once upstream fixes the
# Blocked mapping -- and not merely because it has been quiet for a while.
# drivers/citrineos/README.md carries the full gap table.
e2e-citrineos:
runs-on: ubuntu-latest
needs: check
continue-on-error: true
timeout-minutes: 45
env:
CSMS_DRIVER: ./drivers/citrineos/index.ts
# localhost, not the compose service name -- same asymmetry as the SteVe
# job: the runner reaches the message API through the published port,
# while the simulator containers reach the OCPP endpoint by service name
# from inside the network.
CITRINE_API_URL: http://localhost:8080
OCPP_CP_IDS: CERTCP1,CERTCP2,CERTCP3
steps:
- uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0
with:
persist-credentials: false

# Before the toolchain and without --wait, for the same reason as the
# SteVe job: pulling the images and running the migrations on a cold
# database shares nothing with installing bun.
- name: Start CitrineOS
run: docker compose -f drivers/citrineos/compose.yaml up -d

- uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
with:
bun-version: ${{ env.BUN_VERSION }}

- run: bun install --frozen-lockfile

- name: Wait for it to be healthy
run: docker compose -f drivers/citrineos/compose.yaml up -d --wait

- name: Seed the fixtures the scenarios assume
run: bun bin/ocpp-tck.ts driver provision

# Same reasoning as the SteVe job: a verb nothing runs is a verb that rots.
- name: Environment matches the scenarios
run: bun bin/ocpp-tck.ts driver verify

# `bun run e2e`, exactly as the SteVe job runs it -- both sweeps, one
# script, so the two jobs cannot drift apart and neither can drift from
# what a human runs by hand. Seven scenarios are NOT APPLICABLE here and
# never start a container, so this is cheaper than SteVe's rather than
# doubling the bill.
- name: Conformance sweep
run: bun run e2e

- name: Upload results
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: ocpp-tck-citrineos-results-${{ github.run_id }}
path: results/
if-no-files-found: warn

- name: Tear down
if: always()
run: docker compose -f drivers/citrineos/compose.yaml down -v
22 changes: 21 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,9 +151,29 @@ Put it on the **module**, not inside `create()`. `check-driver` and the
preflight read it without calling `create()`, which is what lets them run with
no credentials — including in your CI, which has none.

### Two worked examples

Two drivers ship here, and they are worth reading together because they solve
the same problem through completely different surfaces — which is the strongest
evidence that the contract is not shaped around either of them.

`drivers/steve/` drives operations through an HTML manager UI and reads state
from MariaDB. `drivers/citrineos/` posts JSON to a generated REST API and reads
from Postgres, and additionally supports two incompatible releases of its CSMS
behind one `CITRINE_VARIANT` setting.

For the mechanics of the contract — the operation switch, the record queries,
the bootstrap verbs — either one works as a model. For **what to do when your
CSMS cannot do something**, read `drivers/citrineos/`: it is the one with gaps,
so it is the one that shows a scope table with `NOT_APPLICABLE` rows each citing
the endpoint that does not exist, a `records` object that omits `reservations`
outright rather than faking it, and a
[README](drivers/citrineos/README.md) whose gap table names a source fact per
row.

## Contributing back

Pull requests welcome for the core, the reference driver, and new scenarios.
Pull requests welcome for the core, either bundled driver, and new scenarios.
Two rules the test suite enforces rather than merely documents:

- **`tests/generic-core.sh`** — nothing under `tck/` may name a CSMS, and the
Expand Down
46 changes: 38 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,17 @@ the OCPP-J frames both directions, and asserts on the captured wire log. What
it does *not* bring is any knowledge of your CSMS: telling it "reset this
charge point" is a **driver**, and a driver is one file you write.

The scenarios trace to the Open Charge Alliance's OCPP 1.6 certification
material — the [OCPP 1.6 certification page][octt], and in particular its
*Test Procedure & Test Plans* and *OCPP Compliancy Testing Tool — Test Case
Document*. Scenario `cert16-tcNNN-…` corresponds to OCA test case `TC_NNN`,
and because the CSMS is the system under test here, the **`_CSMS` variant** is
the one that applies: it is the Central System that must answer. This is not a
certification tool and passing it is not certification — but the reference is
where a disagreement about what a scenario *should* assert gets settled.

[sim]: https://github.com/shiv3/ocpp-cp-simulator
[octt]: https://openchargealliance.org/certificationocpp/certification-ocpp-1-6/

```
your CSMS <--- REST/UI/SQL --- your driver ---.
Expand All @@ -31,10 +41,29 @@ green one.
Node cannot run it, and `bin/ocpp-tck.ts` says so rather than failing later.
- docker — one simulator container per scenario.

## Quick start, against the reference driver
## Bundled drivers

Two ship here, on equal footing. They reach their CSMS through completely
different surfaces, and each answers a question the other cannot:

| Driver | Transport | What it answers | Result |
|---|---|---|---|
| [`drivers/steve`](drivers/steve) | HTML manager UI + MariaDB | *Has the harness lost a capability?* SteVe is the CSMS the scenarios were originally written against, so a scope row that had to be demoted would mean the core dropped something. | All 47 `DRIVABLE` |
| [`drivers/citrineos`](drivers/citrineos/README.md) | JSON REST API + Postgres | *Is the contract actually CSMS-neutral?* [CitrineOS](https://github.com/citrineos/citrineos-core) (LF Energy / S44) had no part in writing the scenarios and has a smaller OCPP 1.6 surface. | 38 `PASS`, 7 `NOT APPLICABLE`, 2 `FAIL` |

An abstraction with one implementation is neutral by assertion, so the second
driver is what turns that into a measurement — and the result is the useful
part: 38 scenarios pass unmodified against a CSMS that had no part in writing
them, 7 report a capability it does not have for OCPP 1.6, and 2 stay red
because they found something. That two drivers this different need no change to
a single scenario is the claim the pair exists to support.

## Quick start, against SteVe

SteVe is the CSMS these scenarios were originally written against, so its
driver ships here as the reference implementation.
Either bundled driver will do; SteVe is shown here because its environment is a
single container pair.
[`drivers/citrineos/README.md`](drivers/citrineos/README.md) is the equivalent
walkthrough for the other one.

```sh
bun add open-ocpp-tck # or: bun add github:juherr/open-ocpp-tck#v0.1.0
Expand Down Expand Up @@ -111,10 +140,10 @@ re-sync, and your driver can live in a completely different repository.
| `ocpp-tck print-sim-image` | nothing | The pinned simulator image digest |
| `ocpp-tck driver <verb>` | driver-defined | A bootstrap verb your driver contributes |

The SteVe driver contributes three: `provision` (seed the fixtures, idempotent),
`verify` (read-only — are they there?), `teardown` (remove them). A driver that
needs no bootstrap contributes none, and the runner never calls them during a
scenario.
Both bundled drivers contribute the same three: `provision` (seed the fixtures,
idempotent), `verify` (read-only — are they there?), `teardown` (remove them).
A driver that needs no bootstrap contributes none, and the runner never calls
them during a scenario.

`check-driver` reads the driver **module** and never calls `create()`, so it
works with no credentials — which is the same property that lets a scenario
Expand Down Expand Up @@ -177,7 +206,8 @@ even where the code is new.
```sh
bun install
bun run typecheck # tsc --noEmit
bun run check:reference-driver # offline
bun run check:steve-driver # offline
bun run check:citrineos-driver # offline
bun test # vendor integrity, genericity, spec invariants
bash tests/types-current.sh # committed declarations match the sources
bash tools/vendor-diff.sh # network: how far upstream has moved
Expand Down
Loading
Loading