diff --git a/.agents/skills/skill-lifecycle/SKILL.md b/.agents/skills/skill-lifecycle/SKILL.md index 747a6c2fd..4a8aa6f89 100644 --- a/.agents/skills/skill-lifecycle/SKILL.md +++ b/.agents/skills/skill-lifecycle/SKILL.md @@ -46,7 +46,7 @@ A skill surfaces at a trigger moment. A rule that binds every action all the tim Packaging keeps one topic in one authoritative place while the skill makes it surface automatically. It has three shapes, and each pairing states which it uses: - **Moved content.** The law-doc section keeps a summary and the skill holds the full rules (`git-commit-conventions`, `comment-and-doc-style`, `pr-review-conduct`). The section ends with the standard pointer sentence: packaged as the named skill at `.agents/skills//SKILL.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, read the skill for the full rules. -- **Kept authority.** The source doc keeps the full rules and the skill is the summary that routes to them (`audit-a-repo` over `AUDIT.md`, `workflow-ci-contract` over `WORKFLOW.md` outside section 4). The skill states per topic which doc section owns it. -- **Included content.** The doc keeps the full rules and the skill needs them whole to work in isolation, so it carries the section as a generated include rather than as a summary or a copy, declared with the region markers "The Pipeline" above describes and keyed on the doc's section (`agent-conduct` over the three `GOVERNANCE.md` sections it surfaces, `workflow-ci-contract` over `WORKFLOW.md` section 4). The doc side states the shape with one sentence naming the skill that includes the section, and the skill side is the region itself. A section carried this way is read outside its own document, so it names a sibling section by document and heading rather than as above or below, and it links to no file by a relative path, since the path would resolve against the skill's directory rather than the doc's. The doc wins by construction, since `scripts/build_dist.py` writes the region from it and its `--check` reports a region that differs from it as stale. +- **Kept authority.** The source doc keeps the full rules and the skill is the summary that routes to them (`audit-a-repo` over `AUDIT.md`, `workflow-ci-contract` over `WORKFLOW.md` outside sections 3, 4, and 5). The skill states per topic which doc section owns it. +- **Included content.** The doc keeps the full rules and the skill needs them whole to work in isolation, so it carries the section as a generated include rather than as a summary or a copy, declared with the region markers "The Pipeline" above describes and keyed on the doc's section (`agent-conduct` over the three `GOVERNANCE.md` sections it surfaces, `workflow-ci-contract` over `WORKFLOW.md` sections 3, 4, and 5). The doc side states the shape with one sentence naming the skill that includes the section, and the skill side is the region itself. A section carried this way is read outside its own document, so it names a sibling section by document and heading rather than as above or below, and it links to no file by a relative path, since the path would resolve against the skill's directory rather than the doc's. The doc wins by construction, since `scripts/build_dist.py` writes the region from it and its `--check` reports a region that differs from it as stale. In every shape the doc is the authority when the two are found to disagree, the moved-content shape included: the doc's summary says what the rule is, and the skill's full text is what gets corrected. A deliberate change to a packaged rule is not such a disagreement. It lands where the full text lives, and in the same change the author either edits the other side's summary to match, since a summary has no mechanical check, or regenerates the include, which has one. A rule stated fully in both places by hand is the drift this pattern exists to prevent, and an include is the one full second statement that cannot drift undetected. diff --git a/.agents/skills/workflow-ci-contract/SKILL.md b/.agents/skills/workflow-ci-contract/SKILL.md index c761fb838..8f7514137 100644 --- a/.agents/skills/workflow-ci-contract/SKILL.md +++ b/.agents/skills/workflow-ci-contract/SKILL.md @@ -8,31 +8,23 @@ description: >- ## Why This Exists -`WORKFLOW.md` is the fleet's CI/CD behavioral contract. This skill is that contract's surface, so an agent editing workflow YAML has the contract in view. It carries the summary, with the contract's section 4 carried whole as a generated include and the test methodology indexed in `references/`. `WORKFLOW.md`'s own canonical-scope note says which of it and `GOVERNANCE.md` is authoritative where the two overlap. +`WORKFLOW.md` is the fleet's CI/CD behavioral contract. This skill is that contract's surface, so an agent editing workflow YAML has the contract in view. It carries the summary, and `WORKFLOW.md` sections 3, 4, and 5 are each carried whole in `references/` as a generated include. `WORKFLOW.md`'s own canonical-scope note says which of it and `GOVERNANCE.md` is authoritative where the two overlap. ## How the Contract Is Read - **Outcomes, not bytes.** A workflow is judged against `WORKFLOW.md` section 4's expected inputs and outputs, never against a snippet byte for byte, per `GOVERNANCE.md` "Foundational Principles". -- **Applicability.** A guarantee governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. +- **Applicability.** A guarantee, or a 5A check or 5B scenario from `WORKFLOW.md` section 5, governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. - **Operational is binary.** Every applicable guarantee holds, or the workflow is not operational. A single applicable input-output mismatch is a defect regardless of how clean the YAML looks. -- **Reached, not carried.** A standard workflow whose job graph is identical across repos of a type is reached as a hub-hosted `workflow_call` task, per `GOVERNANCE.md` "Hub-Hosted Tooling". The repo's own surface is the caller stub, pinned to a hub release commit, and a composite-action hook at `.github/actions/` for what is its own. A hub task reaches its own actions and sibling tasks through `$/`, which resolves at that pinned commit. The merge-bot is the first, and `docs/reusable-workflows.md` in the hub carries the model, the hook contract, and the phase each workflow migrates in. Until a workflow's phase ships, its copy is graded as below. -- **Two layers.** Orchestration (the PR entry workflow, publisher, version and release jobs) is generic and standard at the job level. Build leaves (the `build-` tasks) are repo-owned. Inputs like `github`/`dockerhub`/`expect_release_assets` live on the orchestrator, a leaf receives `ref`/`branch`/`smoke` and whatever else its target needs, a derived `push` among them where that leaf pushes, so assert each input in the layer that declares it. A package target declares no push input on either layer, its push living in a separate `publish-` job in the repo's own publisher. What a repo curates is the list of targets, and adding or dropping one edits the whole surface together: the `enable_` input, the `build-` job and its `github-release` and `build-docker` `needs:` entries, the `changes` paths-filter entry and output, the `smoke-build` enable-forward, and a package target's `publish-` job (D6.4). +- **Reached, not carried.** A standard workflow whose job graph is identical across repos of a type is reached as a hub-hosted `workflow_call` task, per `GOVERNANCE.md` "Hub-Hosted Tooling". The repo's own surface is the caller stub, pinned to a hub release commit, and a composite-action hook at `.github/actions/` for what is its own. A hub task reaches its own actions and sibling tasks through `$/`, which resolves at that pinned commit. The merge-bot is the first, and `docs/reusable-workflows.md` in the hub carries the model, the hook contract, and the stage each workflow migrates in. Until a workflow's stage ships, its copy is graded against the same contract. +- **Two layers.** The pipeline splits into an orchestrator layer and a build-leaf layer, defined in `WORKFLOW.md` section 3's `Two Layers: Orchestration vs Build` and carried in `references/architecture.md`, while `WORKFLOW.md` section 1's `Two layers when auditing` maps which layer declares which input. When a check names an input, assert it in the layer that declares it. ## Style Rules `GOVERNANCE.md` "Workflow YAML Conventions" keeps the style rules, and the `comment-and-doc-style` Skill keeps the line-ending policy, reached from `GOVERNANCE.md` "Documentation Style Conventions" under "Line Endings". Read both before editing a workflow or a composite action. -## The Core Behavioral Spine +## The Contract Text -- **PRs validate fast and never publish**: a paths-filter smoke-builds only changed targets, the caller's own job reaching the reusable validator, or the replacement it points its aggregator at, always runs, and one required aggregator gates the merge, running under `if: always()` so a failed or skipped dependency cannot skip the gate itself, treating skipped smoke as pass and blocking on failure or cancelled. Smoke does a full compile/lint/test but pushes nothing and uploads nothing, every `upload-artifact` gated on smoke being false, which is `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings. -- **A human merge never auto-publishes**, per `GOVERNANCE.md` "Release Model": a `plan` job decides once and every job gates on it. Publishes come from a code-affecting bot push to `main`, a manual dispatch of `main` or `develop`, or the main-only weekly Docker schedule, while a publisher whose only trigger is `workflow_dispatch` (`releaseTrigger: dispatch-only`) reaches the dispatch alone, its bot-push and schedule paths never firing, which covers a source-only repo and an operational repo alike. Each run builds the one trigger branch, the default branch a clean `X.Y.Z`, anything else a prerelease `X.Y.Z-g`, with NBGV owning the patch from git height. The gate's branch literal, the `prerelease` expression, and `version.json`'s `publicReleaseRefSpec` each name the repo's actual default branch, and a divergence among the three is a defect. The release tags the built commit's SHA (`GitCommitId`), never a branch name. -- **Validate at entry**: `GOVERNANCE.md` "Workflow YAML Conventions" keeps the rule. The release gate that applies it checks branch-versus-prerelease in both directions, strips `+buildmetadata`, and on smoke skips the check while the job still succeeds. -- **The seam contract**: a target contributes a release file by uploading `release-asset--`, and the release job collects by `pattern:` plus `merge-multiple:`, never `artifact-ids:`, canonical even for a single target. A caller with no file target passes `expect_release_assets: false`, which covers a Docker-only, a PyPI-only, and a source-only repo, while a NuGet-only caller keeps the default `true`, its leaf uploading a `release-asset-*` that carries the package. -- **Artifacts are an intra-run handoff**: `GOVERNANCE.md` "Workflow YAML Conventions" keeps the consume-then-delete rule, the condition that gates the delete, and the blanket-delete ban. D5 in `references/d-guarantees.md` states the outputs a pipeline shows for them. -- **No-op republish**: an unchanged version re-pushes nothing, the release-create step skips when the tag exists and is refreshed only on a dispatch, registries dedupe server-side (`--skip-duplicate`, `skip-existing: true`), and Docker alone always re-pushes by design. -- **A build failure blocks every publish target**: `github-release` needs every build and guards with `!failure() && !cancelled()` as the terminal registry pusher (Docker) does, and a package target's separate `publish-` job `needs:` the release-task call, so no build failure ships anything partial. A failed package push is outside that, since it runs after the release is cut. - -`references/d-guarantees.md` carries `WORKFLOW.md` section 4 whole as a generated include, so a guarantee's exact wording is one read away. `references/test-methodology.md` indexes `WORKFLOW.md` section 5's audit, trace, and probe procedure, and the sweep itself is run from section 5, which carries the whole core list, the per-type addenda, and the scenario table. +`references/architecture.md`, `references/d-guarantees.md`, and `references/test-methodology.md` carry `WORKFLOW.md` sections 3, 4, and 5 whole, each as a generated include, so the pipeline's architecture, a guarantee's exact wording, and the audit-trace-probe procedure are each one read away rather than restated in full here. A defect in an include region is fixed in `WORKFLOW.md` and regenerated, never edited in this skill, per the `skill-lifecycle` Skill. `WORKFLOW.md` keeps sections 1, 2, and 6 itself, the applicability rule, the style-rule pointer, and the per-project-type walkthroughs that say which items go N/A per type, so read those there. ## After Any Workflow Edit diff --git a/.agents/skills/workflow-ci-contract/references/architecture.md b/.agents/skills/workflow-ci-contract/references/architecture.md new file mode 100644 index 000000000..cfd6598a2 --- /dev/null +++ b/.agents/skills/workflow-ci-contract/references/architecture.md @@ -0,0 +1,112 @@ +# The Pipeline Architecture + +The section below is `WORKFLOW.md` section 3, whole. The D-guarantees it cites by number are `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. + +## The Architecture + + + +### Branch Model + +Two workflow models, set per repo by the registry `workflowModel` field. `release` (default) is the feature-branch pipeline `WORKFLOW.md` specifies: + +```mermaid +flowchart LR + feature[feature branch] -->|squash| develop + develop -->|merge commit| main + main -.->|no back-merge| develop +``` + +`operational` repos (live-service config, `workflowModel: operational`) commit directly to `develop` and promote a known-good snapshot to `main` via an occasional PR: + +```mermaid +flowchart LR + edit[direct signed commit] -->|advisory CI| develop + pr[pull request] -->|lint CI, reported not required| develop + develop -->|merge commit, enforced lint CI| main +``` + +The direct commit is an **allowance, not a substitute for review**. The ruleset drops the pull-request *requirement*, which permits a direct push without withdrawing the pull request, so a change worth reviewing still takes one and both paths reach `develop` legally. Which changes those are is stated as a shape rather than a line count in `GOVERNANCE.md` "Operational Repositories", which owns the test and is the one place it is written, since nothing in a ruleset can apply it. What differs is when validation lands. On the direct-commit path the commit is already on the branch, so CI can only be advisory after the fact, and that is the accepted cost of the model. On the pull-request path the change has not landed, so validation is pre-merge and actionable, which is the moment it is worth the most, and the lint workflow's `pull_request` trigger therefore names `develop` alongside `main` (`WORKFLOW.md` section 6). That is what makes **D1.2** hold here, since its input is *any* PR and the operational model is no exception. The check is reported on a `develop` PR rather than required, because a required status check on `develop` binds the direct push too and would dissolve the allowance the model is built on. + +Their CI is lint/validation only (editorconfig/EOL plus domain linters such as Home Assistant or ESPHome config validation or a firmware build, but **no unit tests**), so the D-guarantees in `WORKFLOW.md` section 4 that assume a build/test pipeline are **N/A** exactly as for `source-only` (`WORKFLOW.md` section 6). What binds: the promotion gate, where the `develop -> main` PR must pass the required `Check pull request workflow status job`, and the source-only release on manual dispatch (`releaseTrigger: dispatch-only`; tag + source zip). Branch-model rulesets are specified in `GOVERNANCE.md` "Branching Model" rather than in `WORKFLOW.md`. + +### Two Layers: Orchestration vs Build + +- **Orchestration** is generic and forms the standardization baseline **at the job level**: the single-branch publisher, the `get-version`, `validate-release`, and `github-release` jobs, and the `changes -> smoke-build -> aggregator` shape of the PR workflow. These job *bodies* should not need per-repo edits. +- **Build** is repo-owned in shape: the `build-` leaf tasks, whether this repo hosts them itself or reaches hub-hosted ones by pin. +- **What the repo curates** (by design, not a leak): the *list* of targets. This is **not** a byte-for-byte file carry. Adding or dropping a target edits the orchestrator's surface: the `enable_` inputs and the `build-` job + its `github-release` **and** `build-docker` `needs:` entries in the release task, **and** the `changes` paths-filter entry + output + the `smoke-build` enable-forward in the PR workflow, plus the separate `publish-` job for a package target. "Verbatim" applies to the `github-release` job and the version/publish-plan logic, not to the release task's job list or the paths-filter. Subsetting is symmetric: the same surface you trim to drop a target you extend to add a new one (e.g. a `release-asset--library` producer needs a new `enable_library` input, a `build-library` job, its two `needs:` entries, and a `library` paths-filter entry, output, and `smoke-build` enable-forward). + +### The Seam Contract + +A target contributes a file to the GitHub release by uploading a workflow artifact named `release-asset--`. The release job collects **every** matching artifact by **pattern** (`pattern: release-asset--*` + `merge-multiple: true`), never an `artifact-ids:` naming one job's output. Canonical for **every** repo, single-target included. Switching to an `artifact-id` handoff forks the release download and breaks the verbatim carry. + +```mermaid +flowchart LR + dotnet[dotnet-publish] -->|release-asset-BRANCH-dotnet-publish| store[(run artifacts)] + nuget[build-nuget] -->|release-asset-BRANCH-nuget| store + store -->|pattern + merge-multiple| rel["github-release job (D6)"] + nuget -->|nuget-build-BRANCH| pub["publish-TARGET job in the repo's own publisher"] + pypi[build-pypi] -->|pypi-build-BRANCH| pub + pub -->|push| registries[(registries)] + docker[build-docker] -->|push| registries +``` + +The diagram writes `BRANCH` and `TARGET` where the prose writes `` and ``, because a mermaid label is sanitized as HTML at render and an angle-bracket placeholder is dropped as an unknown tag. This reaches node labels as well as edge labels, which is why the Release Model diagram below writes `X.Y.Z-g-sha` rather than bracketing its own placeholder. + +### Reusable-Task Parameter Contract + +Every leaf and the release task take `ref`, `branch` (the **logical** branch that drives config/tags/prerelease), and where relevant `smoke`. Branch-derived config keys off `inputs.branch` (the logical branch the caller passes). Artifact names are branch-suffixed. + +### Versioning + +NBGV versions the branch being published. Each run builds a single branch (the trigger ref), so `GITHUB_REF` already names it and NBGV classifies it directly, and no `IGNORE_GITHUB_REF` override is required. The default branch is the public-release ref, so it builds clean `X.Y.Z`. Every other branch builds a prerelease `X.Y.Z-g`. `version.json`'s `version` is the major.minor floor. NBGV appends the git height as the patch. **NBGV and `version.json` are retained even by a repo with no compiled code**, since they are the source of the release tag (`SemVer2`) and `target_commitish` (`GitCommitId`) and the prerelease classification. The .NET SDK is pulled in only as the versioning toolchain. A package build derives its registry version from the same NBGV outputs, but **not always from `SemVer2`**: the PyPI version is built from `AssemblyFileVersion` (four-part `M.N.P.B`) with a PEP 440 `.dev0` appended on the `develop` branch. A wrapper repo may drive its build/image version from an external committed `name -> version` state file while NBGV still tags the release. + +### Validate-at-Entry + +When a workflow's inputs carry a cross-input or input-versus-derived-state invariant, assert it **once** in a dedicated entry job/step the downstream jobs `needs:`, failing fast with `::error::` before any build or publish. + +### Resource Lifecycle + +Workflow artifacts are an **intra-run handoff** only. Durable copies live on the release/registry. The rule: a transfer artifact handed **between jobs** is deleted by exact name/pattern **at its point of consumption**, the delete is **gated to the half of the consumption whose failure would leave it not yet redundant** (D5.2 names the two halves), and it is **best-effort**. **Every** `upload-artifact` sets `retention-days: 1` as the universal failure-path backstop, so no terminal blanket-delete job is needed. An intermediate consumed only within the same run may rely on the retention backstop alone. The run is **never** blanket-deleted (`.artifacts[].id`). See D5. + +### Fast PR Feedback + +PRs validate fast and never publish: a paths-filter smoke-builds only changed targets. A validation job always runs. Smoke builds compile/lint/test but upload nothing and push nothing. One required aggregator gates the merge. See D1. + +```mermaid +flowchart TD + pr[pull request] --> ch[changes paths-filter] + ch -->|target changed| sb[smoke-build changed targets] + ch -->|workflow-only or docs| skip[smoke-build skipped] + val[validation job] --> agg["Check pull request workflow status job (D1)"] + sb --> agg + skip --> agg + agg -->|success| ok[merge allowed] +``` + +### Release Model + +Each publish builds a **single branch**, the trigger ref (`main` a release, `develop` a prerelease), so there is no branch matrix and `github.ref` always names the built branch. A **human merge never auto-publishes**: a first `plan` job (`publish-plan-task.yml`) decides once and every job gates on it. A run publishes on a **code-affecting bot push to `main`** (the App merges every Dependabot/codegen PR, so `github.actor` gates it, and a shared paths filter also drops a non-substantive change like an Actions bump), a **manual dispatch** of `main`/`develop`, or a **main-only weekly schedule** (Docker, to refresh the base image). The `push` is main-only, so a develop bot merge publishes nothing (its prerelease comes via dispatch). A **source-only** repo publishes on **dispatch only**. Every release is a tag on the built commit plus a source archive, README, and LICENSE. Targets amend it with `release-asset-*` files, and a registry push contributes none, made by the Docker leaf for an image and by the separate `publish-` job for a package. An unchanged version re-pushes nothing (no-op republish). Docker re-pushes by design. + +```mermaid +flowchart TD + trig[main-only schedule / dispatch / paths-filtered push] --> one[build the one trigger branch] + one -->|main| vmain["version X.Y.Z stable (D3)"] + one -->|develop| vdev["version X.Y.Z-g-sha prerelease (D3)"] + vmain --> relm["github-release + registries: latest (D4)"] + vdev --> reld["github-release + registries: prerelease (D4)"] +``` + +### Output Seam by Destination + +Pick each output's path by **where the artifact goes**: + +- **File on the GitHub release** (zip, binary, packaged library): one leaf per output uploading `release-asset--`. The repo keeps `expect_release_assets: true` (its default). +- **Package-registry push** (NuGet, PyPI): the leaf builds and uploads a build artifact (`nuget-build-` / `pypi-build-`), and a separate `publish-` job in the **publishing repository's own** publisher consumes it and pushes. Both registries publish through OIDC Trusted Publishing, never a stored API key, and two things put that push outside the leaf. Trusted publishing validates the OIDC token's `job_workflow_ref` claim, which names the workflow the job actually ran from, so a push made from a reusable workflow a *different* repository hosts is rejected at the token exchange, NuGet.org answering `HTTP 401` with `does not start with //.github/workflows/`. That alone rules out a leaf another repository hosts. A leaf this repository hosts clears the claim, and the split still applies to it, because a called job declaring no `permissions:` runs under the calling job's whole grant, so a push anywhere inside the release task would put `id-token: write` on every job in it rather than at the one entry point D7.2 requires. The registered trusted-publishing policy therefore names the publisher, `publish-release.yml`. PyPI additionally gates its publish job behind an environment. NuGet.org binds its policy to the workflow file rather than to an environment and needs none. NuGet's leaf also uploads a `release-asset-*` carrying the package, and PyPI contributes none. +- **Image-registry push** (Docker): the leaf pushes the default branch multi-arch (amd64+arm64) and any other branch `amd64`-only (arm64 emulation is reserved for the released image), and contributes no `release-asset-*`. +- **Filesystem on a host the project owns** (a static site, a config tree): the leaf builds the tree, ships it to the host, and contributes no `release-asset-*`. The transport is the repo's own. What the contract fixes is that the deploy is a **separate `workflow_dispatch`** from the release, so a redeploy of an unchanged commit mints no tag and a host rebuild, a rollback, or proving a branch on a non-production environment costs nothing; that its credentials come from a **per-environment GitHub Environment** rather than the repository secret store; and that the deploy ends by asserting **what the host serves** rather than the transport's exit status (D4.6). Retention at the destination is bounded by a declared count with one side recorded as owning the prune, which is the deploy where its credential can observe the destination and the host where that credential is deliberately write-only (D5.6). +- **No file target via the release task** (Docker-only, PyPI-only, source-only): the release is tag + source zip + README + LICENSE. The caller **MUST pass `expect_release_assets: false`** to the release task. A publisher with file targets retains the default `true`. This setting is caller-specific. The default `true` fails on `fail_on_unmatched_files` when no assets exist. A **source-only** repo also passes every `enable_*` input as false because it has no build leaf (see `WORKFLOW.md` section 6). + +`WORKFLOW.md` section 3 keeps the architecture, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/architecture.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. + + diff --git a/.agents/skills/workflow-ci-contract/references/d-guarantees.md b/.agents/skills/workflow-ci-contract/references/d-guarantees.md index f33376a97..f2c626102 100644 --- a/.agents/skills/workflow-ci-contract/references/d-guarantees.md +++ b/.agents/skills/workflow-ci-contract/references/d-guarantees.md @@ -1,6 +1,6 @@ # The D-Guarantees -The section below is `WORKFLOW.md` section 4, whole. Which of its items bind a given repository is `WORKFLOW.md` section 1's applicability rule. +The section below is `WORKFLOW.md` section 4, whole. Which of its items bind a given repository is `WORKFLOW.md` section 1's applicability rule. The architecture these items govern is `WORKFLOW.md` section 3 and the methodology that checks them is `WORKFLOW.md` section 5, carried whole in `architecture.md` and `test-methodology.md` beside this file. ## The Behavioral Contract diff --git a/.agents/skills/workflow-ci-contract/references/test-methodology.md b/.agents/skills/workflow-ci-contract/references/test-methodology.md index d1e443cf3..744bff09f 100644 --- a/.agents/skills/workflow-ci-contract/references/test-methodology.md +++ b/.agents/skills/workflow-ci-contract/references/test-methodology.md @@ -1,31 +1,75 @@ # Testing a Repo's Workflows -The three escalating verification modes from `WORKFLOW.md` section 5, which keeps authority. N/A items (a check or scenario for an absent construct) are recorded and excluded, never failed. +The section below is `WORKFLOW.md` section 5, whole. Which of its checks and scenarios bind a given repository is `WORKFLOW.md` section 1's applicability rule, and the D-guarantees each one answers to are `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. -## 5A: Static Audit +## The Test Methodology -Read the workflow files, `version.json`, and whatever else a check names as its own evidence: a project or dependency file, `global.json`, `codecov.yml`, `.gitignore`, the branch ruleset, and the repo's Actions and Dependabot secret names. Assert the structural fact behind each applicable D-guarantee, each pass, fail, or N/A with a `file:line` citation, cite a repository setting by its own name where that setting rather than a file is the evidence, and remember the two layers, asserting each input in the layer that declares it. `WORKFLOW.md` 5A carries the whole core list and the per-type addenda, and the sibling `d-guarantees.md` carries the guarantees each item answers to, so read this as an index into them rather than as the sweep itself. + -The core sweep reaches the paths-filter, naming each target's own build paths so a change touching none marks nothing. It reaches smoke gating on every upload. It reaches the aggregator's `needs:` and its skip and fail handling. It reaches coverage collection and its best-effort Codecov upload in every C# and Python repo that has tests at a profile other than `lint-only`, since a repo whose coverage never reaches Codecov passes every other check in this list. It reaches the entry validation jobs and the two-directional release gate. It reaches the single-branch NBGV classification, with the gate's default-branch literal, the `prerelease` expression, and `version.json`'s `publicReleaseRefSpec` all naming the repo's actual default branch. It reaches `target_commitish` from `GitCommitId`. It reaches the consume-then-delete artifact lifecycle, with `retention-days: 1` everywhere and no blanket delete. It reaches the `pattern:` handoff and `inputs.branch` config. It reaches the publisher's serialized concurrency and the SHA pins. Those are entry points into `WORKFLOW.md` 5A's core list rather than the whole of it. +An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a check or scenario for an absent construct is recorded N/A, not failed. -The per-type addenda cover .NET publish, NuGet, PyPI, Docker, and a static site deployed to a host, several assertions each. Apply only the ones the repo's types imply, and read them in `WORKFLOW.md` 5A rather than from this list. +### 5A. Static Audit (No Execution) -## 5B: Trace Scenarios +Read the workflow files, `version.json`, and whatever else a check below names as its evidence: a project or dependency file, `global.json`, `codecov.yml`, `.gitignore`, the branch ruleset, and the repo's Actions and Dependabot secret names. Assert the structural fact behind each *applicable* D-guarantee, each pass/fail/N-A with a `file:line` citation, and cite a repository setting by its own name where that setting rather than a file is the evidence. Remember the two layers: assert each input in the layer that declares it. -For each applicable scenario, evaluate every job's `if:`/`needs:` against the inputs and compare the predicted run/skip, version, release, and artifact end state to the expected table in `WORKFLOW.md` 5B. The load-bearing ones: +**Core (every repo):** -- **S1** a PR touching a target: that target smoke-builds, nothing uploads, the aggregator succeeds. -- **S5/S6** a bot push to `main`: publishes only when code-affecting, and a human push never does. -- **S7** a publish run builds the one trigger branch with the right classification and leaves no dangling artifacts. -- **S8** a dispatch from a ref other than `main`/`develop` fails fast. -- **S9** a no-op re-run on a schedule or push trigger: release-create skipped, registries dedupe, package build artifacts still deleted, Docker still re-pushes. A dispatch re-run refreshes the release instead. -- **S10** branch and version classification disagree: the gate fails loud and everything downstream skips. -- **S12/S13** a deploy dispatch: ref gate first, environment re-asserted, pointer flip separate, live check names the release, and a production deploy from a non-default ref fails before anything is written. +- **D1:** a `changes` paths-filter job exists wherever the repo has a smoke build, with one entry per target naming the paths that target is built from, so a change touching no target marks nothing (a filter written as a negation instead marks a docs-only change and fails D1.1); the PR entry workflow's smoke call sets every publish flag its release task declares to false (`github`/`dockerhub`, and a package-push flag there is itself a finding, per `WORKFLOW.md` section 1); a pushing leaf receives `smoke: true` and a derived `push` (false on smoke), and a build-only leaf receives `smoke: true` with no `push` to derive; every `upload-artifact` the smoke call reaches, in a build task and in any job collecting other jobs' artifacts alike, is gated on smoke being false, written `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings; the aggregator runs under `if: always()`, `needs:` the validation job, and the `changes` and `smoke-build` jobs too wherever the repo has a smoke build, blocks on `failure`/`cancelled`, and passes on a **skipped smoke build**, so a no-build repo's aggregator, having only the validation job to read, requires that job to have succeeded; the aggregator's own job `name:` is the string the branch ruleset's required-check `context:` carries; a validation job runs unconditionally. +- **D1.6:** the validator the repo's validation job reaches collects coverage and uploads it, in every C# and Python repo that has tests. Its C# leg runs `dotnet test --coverage --coverage-output-format cobertura --results-directory ./coverage`, with `--coverage-output` unset, wherever the repo's test project is MTP-based. An MTP-based repo also ships a root `global.json` declaring the Microsoft.Testing.Platform runner, references `Microsoft.Testing.Extensions.CodeCoverage` at 18.9.0 or later in place of `coverlet.collector`, and carries no `xunit.runner.visualstudio`, while a repo still on the VSTest collector keeps its existing validator pin, a migration owed rather than drift. Its Python leg runs `pytest --cov-report=xml`, and the repo declares `pytest-cov` among its test dependencies, a dev dependency group in a uv project and a `requirements*.txt` entry on pip, selects the coverage source in its own `pyproject.toml` rather than leaving `--cov` unset, and leaves the report at the repo root as `coverage.xml`. Either way the report reaches a `codecov/codecov-action` step made best-effort by `continue-on-error` and/or `fail_ci_if_error: false`, and `CODECOV_TOKEN` is present in **both** the repo's Actions and its Dependabot secret names, the second because a Dependabot-triggered run reads that store and the upload otherwise skips silently on every bot pull request. `codecov.yml` sets the project and patch statuses `informational: true`, or names the threshold the repo enforces instead, and lists any intentionally-untested, non-shipped project under `ignore`. `.gitignore` excludes the coverage output. Record the whole item N/A for a repo with no tests, and for a type the audited repo carries at the `lint-only` profile. +- **D2:** an entry validation job/step exists per complex-input workflow; the release gate checks both directions, strips `+buildmetadata`, and skips on smoke; the publisher rejects a dispatch from a ref other than `main` or `develop`. +- **D3:** each run builds one branch, so NBGV classifies `github.ref` directly (no `IGNORE_GITHUB_REF`), and the default-branch literal in the gate (`== 'main'`), the `prerelease` expression (`!= 'main'`), and `version.json`'s `publicReleaseRefSpec` all name the repo's actual default branch. `version.json` sets the major.minor floor, and NBGV and `version.json` are both retained even by a repo with no compiler, since they own the tag (D3.3). NuGet.org derives the prerelease flag from the SemVer2 `-g` suffix rather than the workflow setting one, and the PyPI version is built from `AssemblyFileVersion` with `.dev0` appended on `develop` only (D3.4). +- **D4:** `target_commitish` is the NBGV commit id; `prerelease` equals `branch != default`; the release-create step is gated `exists == 'false' || github.event_name == 'workflow_dispatch'` (the step output is the string `'false'`, not a boolean); the asset-delete step carries that same condition, narrowed by `inputs.expect_release_assets`. Where `workflow_dispatch` is the publisher's only trigger (`releaseTrigger: dispatch-only`) every run is a dispatch, so the exists-check's skip leg can never fire: record that leg N/A rather than failed, and expect the release-create step to still carry the `exists == 'false' || github.event_name == 'workflow_dispatch'` condition, since D6.4 keeps the `github-release` job body verbatim. A caller with no file target passes `expect_release_assets: false`, which covers a Docker-only, a PyPI-only, and a source-only repo alike, while a NuGet-only caller keeps the default `true` because its leaf uploads a `release-asset-*` carrying the package, and a source-only caller also sets every `enable_*` input false. `github-release` and the terminal registry pusher (Docker) each carry `!failure() && !cancelled()` rather than the implicit `success()`, so a failed build skips both, while a target that merely skipped, being disabled or unchanged, still lets the release and the Docker push proceed, and a package target's separate publish job `needs:` the release-task call for the same reason (D4.5). A first `plan` job decides once whether the run publishes, admitting a code-affecting bot push to `main`, a dispatch of `main` or `develop`, and a `main`-only schedule, and every publishing job gates on that decision (D4.1). +- **D5:** each cross-job transfer artifact has a delete step at its consumer, gated so it runs exactly when the consumption happened rather than when the whole job succeeded (D5.2), `continue-on-error: true`, tolerating a failed listing, and looping all ids; **every** upload sets `retention-days: 1`; **no** cleanup step enumerates and deletes the run's whole artifact set, in whatever jq or API shape it is written. +- **D6:** the release download uses `pattern:`/`merge-multiple:` (no `artifact-ids:`). Branch-derived config reads `inputs.branch` (a `github.ref_name` in such config is a finding). Artifact names are branch-suffixed. The target set is consistent across every surface D6.4 names: the `enable_` input, the `build-` job, that job's entries in the `github-release` and `build-docker` `needs:` lists, the `changes` paths-filter entry **and** its output, the `smoke-build` enable-forward, and any separate `publish-` job the package-registry seam requires. The `inputs.branch` rule above binds a called leaf, while a `publish-` job is in the publisher and reads `github.ref_name` correctly. +- **D7:** the publisher concurrency group is global and ref-independent with `cancel-in-progress: false`. A reusable job declares `permissions:` only where every caller grants that scope at startup, per D7.2. A boolean used by both `workflow_call` and `workflow_dispatch` is declared in both trigger blocks, and a boolean read through `github.event.inputs.` is compared against `'true'` alone, a comparison against the boolean `true` never firing on a string, while an `inputs.` read carries the declared boolean and is used directly, a both-forms comparison there being redundant rather than a finding, so a repo whose booleans all arrive by `workflow_call` records the comparison half N/A. A job or step output is a string too and takes the same `== 'true'` (D7.3). Every cross-job condition that admits a skipped dependency pairs its allowlist with a status-check function such as `always()` or `!failure() && !cancelled()`, since the implicit `success()` is false the moment a `needs:` job skipped (D7.4). +- **D8/D9:** the merge-bot enables auto-merge on `opened`/`reopened` for every Dependabot tier including semver-major, dispatches `--squash`/`--merge` by the PR's base ref, disables on a maintainer-pushed `synchronize`, and keys concurrency on the PR number rather than `github.ref`. Codegen runs as a matrix over both branches, and Dependabot targets both branches with security PRs to the default branch (D8.2). The upstream tracker's `bump-branch-prefix` and `branches` match a merge-bot rule, unless it sets `auto-merge: false`, which prefixes the head so no rule matches (wrapper repos). A gate comparing `github.actor` against hard-coded bot identities emits a `::warning::` on the non-matching branch, and the annotation is optional only where the failure announces itself anyway (D8.4). Actions are SHA-pinned. Names, shells, and conditionals follow `GOVERNANCE.md` "Workflow YAML Conventions", and line endings follow `.editorconfig` (D9.5), per the policy `GOVERNANCE.md` "Documentation Style Conventions" routes to under "Line Endings". -## 5C: Live Probe +**Per-type addenda (apply only the ones present):** -Only for what a static trace cannot settle. Every probe that dispatches a workflow, re-runs a real publish, or acts on the deploy host directly is the maintainer's to run: the agent prepares the command and reads the result back afterwards, and a harness refusal to fire one is the control working, never something to re-shape. The probes are a trivial PR to confirm S1, which runs same-repo only wherever the repo has a Docker leg, since that leg logs in to the registry even on smoke, registry queries after a real publish, the version classification and artifact lifecycle read from a real publish's logs, and the deploy ref gate, which is verified only by tripping it. That gate's evidence is four items, the gate job's conclusion, its error text naming the expected and the received ref, every downstream job recorded skipped rather than passed, and the production environment's deployment list carrying no deployment from the dispatched ref, because a gate that fails open and a gate nobody tripped leave the same empty run history behind. +- **.NET publish:** the smoke runtime set is a strict non-empty subset of the full runtime set. The selected set runs sequentially inside one composite-action job. A non-smoke run uploads one `release-asset--dotnet-publish` artifact, while a smoke run skips the archive and upload steps. +- **NuGet:** `publish-nuget` is a job in the repo's own publisher, never inside the release task and never in a reusable workflow a different repository hosts. `id-token: write` appears on that job only, absent from the build and PR paths, beside `actions: write` for the artifact cleanup. The push uses `--skip-duplicate` and is gated by that job's `needs:` on the release-task call, never on an existence check, so a PR never reaches it. The job consume-then-deletes `nuget-build-` under the download step's own success, per D5.1 and D5.2. `*.nupkg` push also carries the paired `.snupkg` to the symbol server where symbols are enabled. The `release-asset` `.7z` carries the package(s). +- **PyPI:** `publish-pypi` declares `environment: { name: pypi }`. `id-token: write` appears only on that job (absent from the build/PR path). `skip-existing: true` is set on the publish action. The build artifact is deleted under the download step's own success, per D5.2. The `pypi` environment has a deployment-branch rule. +- **Docker:** the leaf reads the external state file for the tag instead of `SemVer2` (wrapper repos only, since a plain Docker repo correctly tags off `SemVer2` and records this N/A). The readme job is gated main-only, both by the caller's branch input and inside the hub-hosted `publish-docker-readme-task.yml` itself. The docker-readme task validates `repositories` XOR `manifest`+`manifest-jq`. The buildcache follows D9.4. +- **Static site deployed to a host:** the generator is pinned by version **and** by a checksum verified before install, declared once across the workflows that install it. The deploy is a dispatch carrying an environment choice, with concurrency keyed on the **environment** and `cancel-in-progress: false`, and production gated to the default branch while any ref may reach a non-production environment. The reusable callee re-asserts the environment name in a job of its own. The upload targets a per-release directory and carries no delete flag at the environment root, and the pointer flip is a separate step. The terminal check asserts the environment, then the release id, then the URL contract, waiting for convergence to a bounded timeout rather than sampling once, and reporting an unreachable host distinctly from an HTTP status (D4.6). Retention is bounded by a declared count and one side is recorded as owning the prune: the deploy asserts it where the credential can observe the destination, and the host owns it where the credential is confined write-only (D5.6). -## Verdict +### 5B. End-to-End Trace Scenarios (No Execution, Deterministic from the YAML) -Record the workflow operational when every applicable 5A item passes, every applicable 5B scenario's predicted output equals the expected, and no 5C probe that was run contradicts either. Any applicable mismatch is a defect. The verdict names the failing guarantees with the triggering input for each, the items recorded N/A, and the 5C probes prepared but not run, so a static-only audit and a fully probed one do not read alike. Per-project-type walkthroughs mapping scenarios onto targets, including source-only, static-site, and operational shapes, are `WORKFLOW.md` section 6. +For each *applicable* scenario, evaluate every job's `if:`/`needs:` against the inputs and emit the predicted **run/skip + version + release + artifact-end-state** table, then compare to the expected. A scenario governing a construct the repo does not contain is N/A, per `WORKFLOW.md` section 1, and an absent trigger is such a construct: a dispatch-only publisher records S5, S6 and S9 N/A, since their push and schedule paths can never fire there. Minimum set: + +| # | Input | Expected output | Exercises | +| --- | --- | --- | --- | +| S1 | PR touching a build target | `changes` flags it; validation runs; that target's smoke build runs; no push, **no uploads**; validate-release **skipped (smoke), succeeds**; release **skipped**; aggregator **success**; version = prerelease; no release; no dangling artifacts | D1, D2.2, D3 | +| S2 | PR changing only docs | smoke-build **skipped**, validation runs, aggregator **success** | D1.1, D1.2, D1.5 | +| S3 | PR changing only `.github/workflows/**` | the filter marks no target -> smoke-build **skipped**, validation runs, aggregator **success** | D1.2, D1.4, D1.5 | +| S4 | PR base = default branch, carrying a build target | smoke versions as prerelease, validate-release **skipped (smoke), succeeds** so the default-branch arm does **not** fire, aggregator **success**, promotion not blocked | D1.5, D2.2, D3.2 | +| S5 | bot push to `main` not touching a release path (e.g. an Actions bump) | the paths filter excludes it, so nothing publishes | D4.1 | +| S6 | code-affecting **bot** push to `main` (a human push/promotion, or any develop push, does not) | the `plan` job gates it to the App/Dependabot actor, and `main` publishes a release | D3, D4 | +| S7 | publish run (schedule, a bot push to main, or a dispatch) | builds the **one** trigger branch: `main` -> `X.Y.Z`, `prerelease=false`, registry stable, readme run; `develop` -> `X.Y.Z-g`, `prerelease=true`, registry prerelease; `release-asset-*` consumed-then-deleted; each package build-artifact (`nuget-build-*`, `pypi-build-*`) deleted after its publish; **no dangling artifacts** | D3, D4, D5, D6, D7 | +| S8 | dispatch from a ref other than `main` or `develop` | **fails fast** | D2.3 | +| S9 | re-run publish on a schedule or push trigger, version unchanged (a dispatch re-run refreshes the release instead, per D4.4) | release-create **skipped**, `release-asset-*` delete **skipped**; NuGet/PyPI pushes no-op (server dedupe); **package build-artifacts still deleted** (their download succeeded); **Docker still re-pushes** the image; no duplicate release | D4.4, D5.2 | +| S10 | branch/version classification disagree | validate-release **fails loud**, build/publish skip | D2.2 | +| S11 | scheduled upstream-version bump (wrapper) | resolver detects a change -> commits the state file -> opens a `-` PR -> merge-bot auto-merges -> the `main` pin publishes via the gate (a develop pin does not auto-publish, shipping instead via a develop dispatch or promotion) | D8.3, D3.5 | +| S12 | deploy dispatch naming an environment | the ref gate runs **first** (production from the default branch only, any ref to a non-production environment); validation runs; the callee re-asserts the environment name; a release installs under its own id; the pointer flips as a separate step; retention is bounded by whichever of the two D5.6 shapes the repo uses, so a deploy whose credential can observe the destination asserts the count converged and one confined write-only leaves it to the host; the live check asserts the environment and the release id, waiting out the reload, then the URL contract; **no tag and no release are created** | D2.1, D4.6, D5.6 | +| S13 | deploy dispatch of a production environment from a non-default ref | **fails fast**, before anything is installed or written | D2.1 | + +### 5C. Live Probe (Where Warranted) + +Every probe here that dispatches a workflow or re-runs a real publish is the maintainer's to run, and the agent prepares the command and reads the result back afterwards. A harness that refuses such a write is the harness working as intended, and the refusal is neither re-shaped into a raw API call nor talked around (GOVERNANCE.md "Repository Boundaries and Write Safety"). + +- Open a trivial-change PR touching one target and confirm S1. *Caveat: the Docker leg logs in to the registry even on smoke and reads the buildcache, so it needs `DOCKER_HUB_*` secrets and cannot run on a fork PR (same-repo only).* +- Per registry: after a real publish, query NuGet.org for the expected version + prerelease classification (and the `.snupkg` on the symbol server), and confirm a re-run added no duplicate. For PyPI read the built `dist/*` filenames out of the build job's log, `.dev0` off `develop` vs a plain version on the default branch. +- Inspect the latest real publish's logs for `PublicRelease`/`SemVer2` per leg and confirm the artifact lifecycle (uploaded, consumed, deleted, with none left behind). +- **The deploy ref gate (S13) is verified only by tripping it.** Dispatch the production environment from a non-default ref and expect the run to fail at the gate. The evidence is four things, and each of them matters: the gate job's conclusion, its error text naming the expected and the received ref, every downstream job recorded as **skipped** rather than passed, and the production environment's deployment list carrying no deployment from the dispatched ref. Capture all four, because a gate that fails open and a gate nobody tripped produce the same empty run history, so "we have never seen it fail" is not evidence about the one control standing between a mis-dispatch and the live site. **The agent prepares the command and reads all four back afterwards. It does not fire it.** The same split applies to any probe that acts on the deploy host directly, an outbound SSH exercising a forced command among them. + +### Assessment + +Record the workflow **operational** when every *applicable* 5A item passes, every *applicable* 5B scenario's predicted output equals the expected, and no 5C probe that was run contradicts either. N/A items are excluded, never counted as failures. Any *applicable* mismatch is a **defect** -> **not operational**. Procedure: + +1. **Audit** with 5A. Record pass/fail/N-A with `file:line`. +2. **Trace** the applicable S-scenarios with 5B. Diff predicted vs expected. +3. **Probe** with 5C where a live signal exists that the static trace cannot produce: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. +4. **Verdict:** operational / not operational, with the failing guarantee(s) and the triggering input for each, the list of items recorded N/A, and the 5C probes prepared but not run. + +`WORKFLOW.md` section 5 keeps the test methodology, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/test-methodology.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. + + diff --git a/.claude-plugin/fleet-skills/.source-digests/skill-lifecycle b/.claude-plugin/fleet-skills/.source-digests/skill-lifecycle index d05a63755..8d070086f 100644 --- a/.claude-plugin/fleet-skills/.source-digests/skill-lifecycle +++ b/.claude-plugin/fleet-skills/.source-digests/skill-lifecycle @@ -1 +1 @@ -8edc99475ac2c84f +644d3f1f0921b41e diff --git a/.claude-plugin/fleet-skills/.source-digests/workflow-ci-contract b/.claude-plugin/fleet-skills/.source-digests/workflow-ci-contract index fe4adb2be..04f6a6b9f 100644 --- a/.claude-plugin/fleet-skills/.source-digests/workflow-ci-contract +++ b/.claude-plugin/fleet-skills/.source-digests/workflow-ci-contract @@ -1 +1 @@ -f3440e45d74297ac +68655ae1cf8f88bf diff --git a/.claude-plugin/fleet-skills/skills/skill-lifecycle/SKILL.md b/.claude-plugin/fleet-skills/skills/skill-lifecycle/SKILL.md index 747a6c2fd..4a8aa6f89 100644 --- a/.claude-plugin/fleet-skills/skills/skill-lifecycle/SKILL.md +++ b/.claude-plugin/fleet-skills/skills/skill-lifecycle/SKILL.md @@ -46,7 +46,7 @@ A skill surfaces at a trigger moment. A rule that binds every action all the tim Packaging keeps one topic in one authoritative place while the skill makes it surface automatically. It has three shapes, and each pairing states which it uses: - **Moved content.** The law-doc section keeps a summary and the skill holds the full rules (`git-commit-conventions`, `comment-and-doc-style`, `pr-review-conduct`). The section ends with the standard pointer sentence: packaged as the named skill at `.agents/skills//SKILL.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, read the skill for the full rules. -- **Kept authority.** The source doc keeps the full rules and the skill is the summary that routes to them (`audit-a-repo` over `AUDIT.md`, `workflow-ci-contract` over `WORKFLOW.md` outside section 4). The skill states per topic which doc section owns it. -- **Included content.** The doc keeps the full rules and the skill needs them whole to work in isolation, so it carries the section as a generated include rather than as a summary or a copy, declared with the region markers "The Pipeline" above describes and keyed on the doc's section (`agent-conduct` over the three `GOVERNANCE.md` sections it surfaces, `workflow-ci-contract` over `WORKFLOW.md` section 4). The doc side states the shape with one sentence naming the skill that includes the section, and the skill side is the region itself. A section carried this way is read outside its own document, so it names a sibling section by document and heading rather than as above or below, and it links to no file by a relative path, since the path would resolve against the skill's directory rather than the doc's. The doc wins by construction, since `scripts/build_dist.py` writes the region from it and its `--check` reports a region that differs from it as stale. +- **Kept authority.** The source doc keeps the full rules and the skill is the summary that routes to them (`audit-a-repo` over `AUDIT.md`, `workflow-ci-contract` over `WORKFLOW.md` outside sections 3, 4, and 5). The skill states per topic which doc section owns it. +- **Included content.** The doc keeps the full rules and the skill needs them whole to work in isolation, so it carries the section as a generated include rather than as a summary or a copy, declared with the region markers "The Pipeline" above describes and keyed on the doc's section (`agent-conduct` over the three `GOVERNANCE.md` sections it surfaces, `workflow-ci-contract` over `WORKFLOW.md` sections 3, 4, and 5). The doc side states the shape with one sentence naming the skill that includes the section, and the skill side is the region itself. A section carried this way is read outside its own document, so it names a sibling section by document and heading rather than as above or below, and it links to no file by a relative path, since the path would resolve against the skill's directory rather than the doc's. The doc wins by construction, since `scripts/build_dist.py` writes the region from it and its `--check` reports a region that differs from it as stale. In every shape the doc is the authority when the two are found to disagree, the moved-content shape included: the doc's summary says what the rule is, and the skill's full text is what gets corrected. A deliberate change to a packaged rule is not such a disagreement. It lands where the full text lives, and in the same change the author either edits the other side's summary to match, since a summary has no mechanical check, or regenerates the include, which has one. A rule stated fully in both places by hand is the drift this pattern exists to prevent, and an include is the one full second statement that cannot drift undetected. diff --git a/.claude-plugin/fleet-skills/skills/workflow-ci-contract/SKILL.md b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/SKILL.md index c761fb838..8f7514137 100644 --- a/.claude-plugin/fleet-skills/skills/workflow-ci-contract/SKILL.md +++ b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/SKILL.md @@ -8,31 +8,23 @@ description: >- ## Why This Exists -`WORKFLOW.md` is the fleet's CI/CD behavioral contract. This skill is that contract's surface, so an agent editing workflow YAML has the contract in view. It carries the summary, with the contract's section 4 carried whole as a generated include and the test methodology indexed in `references/`. `WORKFLOW.md`'s own canonical-scope note says which of it and `GOVERNANCE.md` is authoritative where the two overlap. +`WORKFLOW.md` is the fleet's CI/CD behavioral contract. This skill is that contract's surface, so an agent editing workflow YAML has the contract in view. It carries the summary, and `WORKFLOW.md` sections 3, 4, and 5 are each carried whole in `references/` as a generated include. `WORKFLOW.md`'s own canonical-scope note says which of it and `GOVERNANCE.md` is authoritative where the two overlap. ## How the Contract Is Read - **Outcomes, not bytes.** A workflow is judged against `WORKFLOW.md` section 4's expected inputs and outputs, never against a snippet byte for byte, per `GOVERNANCE.md` "Foundational Principles". -- **Applicability.** A guarantee governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. +- **Applicability.** A guarantee, or a 5A check or 5B scenario from `WORKFLOW.md` section 5, governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. - **Operational is binary.** Every applicable guarantee holds, or the workflow is not operational. A single applicable input-output mismatch is a defect regardless of how clean the YAML looks. -- **Reached, not carried.** A standard workflow whose job graph is identical across repos of a type is reached as a hub-hosted `workflow_call` task, per `GOVERNANCE.md` "Hub-Hosted Tooling". The repo's own surface is the caller stub, pinned to a hub release commit, and a composite-action hook at `.github/actions/` for what is its own. A hub task reaches its own actions and sibling tasks through `$/`, which resolves at that pinned commit. The merge-bot is the first, and `docs/reusable-workflows.md` in the hub carries the model, the hook contract, and the phase each workflow migrates in. Until a workflow's phase ships, its copy is graded as below. -- **Two layers.** Orchestration (the PR entry workflow, publisher, version and release jobs) is generic and standard at the job level. Build leaves (the `build-` tasks) are repo-owned. Inputs like `github`/`dockerhub`/`expect_release_assets` live on the orchestrator, a leaf receives `ref`/`branch`/`smoke` and whatever else its target needs, a derived `push` among them where that leaf pushes, so assert each input in the layer that declares it. A package target declares no push input on either layer, its push living in a separate `publish-` job in the repo's own publisher. What a repo curates is the list of targets, and adding or dropping one edits the whole surface together: the `enable_` input, the `build-` job and its `github-release` and `build-docker` `needs:` entries, the `changes` paths-filter entry and output, the `smoke-build` enable-forward, and a package target's `publish-` job (D6.4). +- **Reached, not carried.** A standard workflow whose job graph is identical across repos of a type is reached as a hub-hosted `workflow_call` task, per `GOVERNANCE.md` "Hub-Hosted Tooling". The repo's own surface is the caller stub, pinned to a hub release commit, and a composite-action hook at `.github/actions/` for what is its own. A hub task reaches its own actions and sibling tasks through `$/`, which resolves at that pinned commit. The merge-bot is the first, and `docs/reusable-workflows.md` in the hub carries the model, the hook contract, and the stage each workflow migrates in. Until a workflow's stage ships, its copy is graded against the same contract. +- **Two layers.** The pipeline splits into an orchestrator layer and a build-leaf layer, defined in `WORKFLOW.md` section 3's `Two Layers: Orchestration vs Build` and carried in `references/architecture.md`, while `WORKFLOW.md` section 1's `Two layers when auditing` maps which layer declares which input. When a check names an input, assert it in the layer that declares it. ## Style Rules `GOVERNANCE.md` "Workflow YAML Conventions" keeps the style rules, and the `comment-and-doc-style` Skill keeps the line-ending policy, reached from `GOVERNANCE.md` "Documentation Style Conventions" under "Line Endings". Read both before editing a workflow or a composite action. -## The Core Behavioral Spine +## The Contract Text -- **PRs validate fast and never publish**: a paths-filter smoke-builds only changed targets, the caller's own job reaching the reusable validator, or the replacement it points its aggregator at, always runs, and one required aggregator gates the merge, running under `if: always()` so a failed or skipped dependency cannot skip the gate itself, treating skipped smoke as pass and blocking on failure or cancelled. Smoke does a full compile/lint/test but pushes nothing and uploads nothing, every `upload-artifact` gated on smoke being false, which is `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings. -- **A human merge never auto-publishes**, per `GOVERNANCE.md` "Release Model": a `plan` job decides once and every job gates on it. Publishes come from a code-affecting bot push to `main`, a manual dispatch of `main` or `develop`, or the main-only weekly Docker schedule, while a publisher whose only trigger is `workflow_dispatch` (`releaseTrigger: dispatch-only`) reaches the dispatch alone, its bot-push and schedule paths never firing, which covers a source-only repo and an operational repo alike. Each run builds the one trigger branch, the default branch a clean `X.Y.Z`, anything else a prerelease `X.Y.Z-g`, with NBGV owning the patch from git height. The gate's branch literal, the `prerelease` expression, and `version.json`'s `publicReleaseRefSpec` each name the repo's actual default branch, and a divergence among the three is a defect. The release tags the built commit's SHA (`GitCommitId`), never a branch name. -- **Validate at entry**: `GOVERNANCE.md` "Workflow YAML Conventions" keeps the rule. The release gate that applies it checks branch-versus-prerelease in both directions, strips `+buildmetadata`, and on smoke skips the check while the job still succeeds. -- **The seam contract**: a target contributes a release file by uploading `release-asset--`, and the release job collects by `pattern:` plus `merge-multiple:`, never `artifact-ids:`, canonical even for a single target. A caller with no file target passes `expect_release_assets: false`, which covers a Docker-only, a PyPI-only, and a source-only repo, while a NuGet-only caller keeps the default `true`, its leaf uploading a `release-asset-*` that carries the package. -- **Artifacts are an intra-run handoff**: `GOVERNANCE.md` "Workflow YAML Conventions" keeps the consume-then-delete rule, the condition that gates the delete, and the blanket-delete ban. D5 in `references/d-guarantees.md` states the outputs a pipeline shows for them. -- **No-op republish**: an unchanged version re-pushes nothing, the release-create step skips when the tag exists and is refreshed only on a dispatch, registries dedupe server-side (`--skip-duplicate`, `skip-existing: true`), and Docker alone always re-pushes by design. -- **A build failure blocks every publish target**: `github-release` needs every build and guards with `!failure() && !cancelled()` as the terminal registry pusher (Docker) does, and a package target's separate `publish-` job `needs:` the release-task call, so no build failure ships anything partial. A failed package push is outside that, since it runs after the release is cut. - -`references/d-guarantees.md` carries `WORKFLOW.md` section 4 whole as a generated include, so a guarantee's exact wording is one read away. `references/test-methodology.md` indexes `WORKFLOW.md` section 5's audit, trace, and probe procedure, and the sweep itself is run from section 5, which carries the whole core list, the per-type addenda, and the scenario table. +`references/architecture.md`, `references/d-guarantees.md`, and `references/test-methodology.md` carry `WORKFLOW.md` sections 3, 4, and 5 whole, each as a generated include, so the pipeline's architecture, a guarantee's exact wording, and the audit-trace-probe procedure are each one read away rather than restated in full here. A defect in an include region is fixed in `WORKFLOW.md` and regenerated, never edited in this skill, per the `skill-lifecycle` Skill. `WORKFLOW.md` keeps sections 1, 2, and 6 itself, the applicability rule, the style-rule pointer, and the per-project-type walkthroughs that say which items go N/A per type, so read those there. ## After Any Workflow Edit diff --git a/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/architecture.md b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/architecture.md new file mode 100644 index 000000000..cfd6598a2 --- /dev/null +++ b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/architecture.md @@ -0,0 +1,112 @@ +# The Pipeline Architecture + +The section below is `WORKFLOW.md` section 3, whole. The D-guarantees it cites by number are `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. + +## The Architecture + + + +### Branch Model + +Two workflow models, set per repo by the registry `workflowModel` field. `release` (default) is the feature-branch pipeline `WORKFLOW.md` specifies: + +```mermaid +flowchart LR + feature[feature branch] -->|squash| develop + develop -->|merge commit| main + main -.->|no back-merge| develop +``` + +`operational` repos (live-service config, `workflowModel: operational`) commit directly to `develop` and promote a known-good snapshot to `main` via an occasional PR: + +```mermaid +flowchart LR + edit[direct signed commit] -->|advisory CI| develop + pr[pull request] -->|lint CI, reported not required| develop + develop -->|merge commit, enforced lint CI| main +``` + +The direct commit is an **allowance, not a substitute for review**. The ruleset drops the pull-request *requirement*, which permits a direct push without withdrawing the pull request, so a change worth reviewing still takes one and both paths reach `develop` legally. Which changes those are is stated as a shape rather than a line count in `GOVERNANCE.md` "Operational Repositories", which owns the test and is the one place it is written, since nothing in a ruleset can apply it. What differs is when validation lands. On the direct-commit path the commit is already on the branch, so CI can only be advisory after the fact, and that is the accepted cost of the model. On the pull-request path the change has not landed, so validation is pre-merge and actionable, which is the moment it is worth the most, and the lint workflow's `pull_request` trigger therefore names `develop` alongside `main` (`WORKFLOW.md` section 6). That is what makes **D1.2** hold here, since its input is *any* PR and the operational model is no exception. The check is reported on a `develop` PR rather than required, because a required status check on `develop` binds the direct push too and would dissolve the allowance the model is built on. + +Their CI is lint/validation only (editorconfig/EOL plus domain linters such as Home Assistant or ESPHome config validation or a firmware build, but **no unit tests**), so the D-guarantees in `WORKFLOW.md` section 4 that assume a build/test pipeline are **N/A** exactly as for `source-only` (`WORKFLOW.md` section 6). What binds: the promotion gate, where the `develop -> main` PR must pass the required `Check pull request workflow status job`, and the source-only release on manual dispatch (`releaseTrigger: dispatch-only`; tag + source zip). Branch-model rulesets are specified in `GOVERNANCE.md` "Branching Model" rather than in `WORKFLOW.md`. + +### Two Layers: Orchestration vs Build + +- **Orchestration** is generic and forms the standardization baseline **at the job level**: the single-branch publisher, the `get-version`, `validate-release`, and `github-release` jobs, and the `changes -> smoke-build -> aggregator` shape of the PR workflow. These job *bodies* should not need per-repo edits. +- **Build** is repo-owned in shape: the `build-` leaf tasks, whether this repo hosts them itself or reaches hub-hosted ones by pin. +- **What the repo curates** (by design, not a leak): the *list* of targets. This is **not** a byte-for-byte file carry. Adding or dropping a target edits the orchestrator's surface: the `enable_` inputs and the `build-` job + its `github-release` **and** `build-docker` `needs:` entries in the release task, **and** the `changes` paths-filter entry + output + the `smoke-build` enable-forward in the PR workflow, plus the separate `publish-` job for a package target. "Verbatim" applies to the `github-release` job and the version/publish-plan logic, not to the release task's job list or the paths-filter. Subsetting is symmetric: the same surface you trim to drop a target you extend to add a new one (e.g. a `release-asset--library` producer needs a new `enable_library` input, a `build-library` job, its two `needs:` entries, and a `library` paths-filter entry, output, and `smoke-build` enable-forward). + +### The Seam Contract + +A target contributes a file to the GitHub release by uploading a workflow artifact named `release-asset--`. The release job collects **every** matching artifact by **pattern** (`pattern: release-asset--*` + `merge-multiple: true`), never an `artifact-ids:` naming one job's output. Canonical for **every** repo, single-target included. Switching to an `artifact-id` handoff forks the release download and breaks the verbatim carry. + +```mermaid +flowchart LR + dotnet[dotnet-publish] -->|release-asset-BRANCH-dotnet-publish| store[(run artifacts)] + nuget[build-nuget] -->|release-asset-BRANCH-nuget| store + store -->|pattern + merge-multiple| rel["github-release job (D6)"] + nuget -->|nuget-build-BRANCH| pub["publish-TARGET job in the repo's own publisher"] + pypi[build-pypi] -->|pypi-build-BRANCH| pub + pub -->|push| registries[(registries)] + docker[build-docker] -->|push| registries +``` + +The diagram writes `BRANCH` and `TARGET` where the prose writes `` and ``, because a mermaid label is sanitized as HTML at render and an angle-bracket placeholder is dropped as an unknown tag. This reaches node labels as well as edge labels, which is why the Release Model diagram below writes `X.Y.Z-g-sha` rather than bracketing its own placeholder. + +### Reusable-Task Parameter Contract + +Every leaf and the release task take `ref`, `branch` (the **logical** branch that drives config/tags/prerelease), and where relevant `smoke`. Branch-derived config keys off `inputs.branch` (the logical branch the caller passes). Artifact names are branch-suffixed. + +### Versioning + +NBGV versions the branch being published. Each run builds a single branch (the trigger ref), so `GITHUB_REF` already names it and NBGV classifies it directly, and no `IGNORE_GITHUB_REF` override is required. The default branch is the public-release ref, so it builds clean `X.Y.Z`. Every other branch builds a prerelease `X.Y.Z-g`. `version.json`'s `version` is the major.minor floor. NBGV appends the git height as the patch. **NBGV and `version.json` are retained even by a repo with no compiled code**, since they are the source of the release tag (`SemVer2`) and `target_commitish` (`GitCommitId`) and the prerelease classification. The .NET SDK is pulled in only as the versioning toolchain. A package build derives its registry version from the same NBGV outputs, but **not always from `SemVer2`**: the PyPI version is built from `AssemblyFileVersion` (four-part `M.N.P.B`) with a PEP 440 `.dev0` appended on the `develop` branch. A wrapper repo may drive its build/image version from an external committed `name -> version` state file while NBGV still tags the release. + +### Validate-at-Entry + +When a workflow's inputs carry a cross-input or input-versus-derived-state invariant, assert it **once** in a dedicated entry job/step the downstream jobs `needs:`, failing fast with `::error::` before any build or publish. + +### Resource Lifecycle + +Workflow artifacts are an **intra-run handoff** only. Durable copies live on the release/registry. The rule: a transfer artifact handed **between jobs** is deleted by exact name/pattern **at its point of consumption**, the delete is **gated to the half of the consumption whose failure would leave it not yet redundant** (D5.2 names the two halves), and it is **best-effort**. **Every** `upload-artifact` sets `retention-days: 1` as the universal failure-path backstop, so no terminal blanket-delete job is needed. An intermediate consumed only within the same run may rely on the retention backstop alone. The run is **never** blanket-deleted (`.artifacts[].id`). See D5. + +### Fast PR Feedback + +PRs validate fast and never publish: a paths-filter smoke-builds only changed targets. A validation job always runs. Smoke builds compile/lint/test but upload nothing and push nothing. One required aggregator gates the merge. See D1. + +```mermaid +flowchart TD + pr[pull request] --> ch[changes paths-filter] + ch -->|target changed| sb[smoke-build changed targets] + ch -->|workflow-only or docs| skip[smoke-build skipped] + val[validation job] --> agg["Check pull request workflow status job (D1)"] + sb --> agg + skip --> agg + agg -->|success| ok[merge allowed] +``` + +### Release Model + +Each publish builds a **single branch**, the trigger ref (`main` a release, `develop` a prerelease), so there is no branch matrix and `github.ref` always names the built branch. A **human merge never auto-publishes**: a first `plan` job (`publish-plan-task.yml`) decides once and every job gates on it. A run publishes on a **code-affecting bot push to `main`** (the App merges every Dependabot/codegen PR, so `github.actor` gates it, and a shared paths filter also drops a non-substantive change like an Actions bump), a **manual dispatch** of `main`/`develop`, or a **main-only weekly schedule** (Docker, to refresh the base image). The `push` is main-only, so a develop bot merge publishes nothing (its prerelease comes via dispatch). A **source-only** repo publishes on **dispatch only**. Every release is a tag on the built commit plus a source archive, README, and LICENSE. Targets amend it with `release-asset-*` files, and a registry push contributes none, made by the Docker leaf for an image and by the separate `publish-` job for a package. An unchanged version re-pushes nothing (no-op republish). Docker re-pushes by design. + +```mermaid +flowchart TD + trig[main-only schedule / dispatch / paths-filtered push] --> one[build the one trigger branch] + one -->|main| vmain["version X.Y.Z stable (D3)"] + one -->|develop| vdev["version X.Y.Z-g-sha prerelease (D3)"] + vmain --> relm["github-release + registries: latest (D4)"] + vdev --> reld["github-release + registries: prerelease (D4)"] +``` + +### Output Seam by Destination + +Pick each output's path by **where the artifact goes**: + +- **File on the GitHub release** (zip, binary, packaged library): one leaf per output uploading `release-asset--`. The repo keeps `expect_release_assets: true` (its default). +- **Package-registry push** (NuGet, PyPI): the leaf builds and uploads a build artifact (`nuget-build-` / `pypi-build-`), and a separate `publish-` job in the **publishing repository's own** publisher consumes it and pushes. Both registries publish through OIDC Trusted Publishing, never a stored API key, and two things put that push outside the leaf. Trusted publishing validates the OIDC token's `job_workflow_ref` claim, which names the workflow the job actually ran from, so a push made from a reusable workflow a *different* repository hosts is rejected at the token exchange, NuGet.org answering `HTTP 401` with `does not start with //.github/workflows/`. That alone rules out a leaf another repository hosts. A leaf this repository hosts clears the claim, and the split still applies to it, because a called job declaring no `permissions:` runs under the calling job's whole grant, so a push anywhere inside the release task would put `id-token: write` on every job in it rather than at the one entry point D7.2 requires. The registered trusted-publishing policy therefore names the publisher, `publish-release.yml`. PyPI additionally gates its publish job behind an environment. NuGet.org binds its policy to the workflow file rather than to an environment and needs none. NuGet's leaf also uploads a `release-asset-*` carrying the package, and PyPI contributes none. +- **Image-registry push** (Docker): the leaf pushes the default branch multi-arch (amd64+arm64) and any other branch `amd64`-only (arm64 emulation is reserved for the released image), and contributes no `release-asset-*`. +- **Filesystem on a host the project owns** (a static site, a config tree): the leaf builds the tree, ships it to the host, and contributes no `release-asset-*`. The transport is the repo's own. What the contract fixes is that the deploy is a **separate `workflow_dispatch`** from the release, so a redeploy of an unchanged commit mints no tag and a host rebuild, a rollback, or proving a branch on a non-production environment costs nothing; that its credentials come from a **per-environment GitHub Environment** rather than the repository secret store; and that the deploy ends by asserting **what the host serves** rather than the transport's exit status (D4.6). Retention at the destination is bounded by a declared count with one side recorded as owning the prune, which is the deploy where its credential can observe the destination and the host where that credential is deliberately write-only (D5.6). +- **No file target via the release task** (Docker-only, PyPI-only, source-only): the release is tag + source zip + README + LICENSE. The caller **MUST pass `expect_release_assets: false`** to the release task. A publisher with file targets retains the default `true`. This setting is caller-specific. The default `true` fails on `fail_on_unmatched_files` when no assets exist. A **source-only** repo also passes every `enable_*` input as false because it has no build leaf (see `WORKFLOW.md` section 6). + +`WORKFLOW.md` section 3 keeps the architecture, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/architecture.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. + + diff --git a/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/d-guarantees.md b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/d-guarantees.md index f33376a97..f2c626102 100644 --- a/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/d-guarantees.md +++ b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/d-guarantees.md @@ -1,6 +1,6 @@ # The D-Guarantees -The section below is `WORKFLOW.md` section 4, whole. Which of its items bind a given repository is `WORKFLOW.md` section 1's applicability rule. +The section below is `WORKFLOW.md` section 4, whole. Which of its items bind a given repository is `WORKFLOW.md` section 1's applicability rule. The architecture these items govern is `WORKFLOW.md` section 3 and the methodology that checks them is `WORKFLOW.md` section 5, carried whole in `architecture.md` and `test-methodology.md` beside this file. ## The Behavioral Contract diff --git a/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/test-methodology.md b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/test-methodology.md index d1e443cf3..744bff09f 100644 --- a/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/test-methodology.md +++ b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/test-methodology.md @@ -1,31 +1,75 @@ # Testing a Repo's Workflows -The three escalating verification modes from `WORKFLOW.md` section 5, which keeps authority. N/A items (a check or scenario for an absent construct) are recorded and excluded, never failed. +The section below is `WORKFLOW.md` section 5, whole. Which of its checks and scenarios bind a given repository is `WORKFLOW.md` section 1's applicability rule, and the D-guarantees each one answers to are `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. -## 5A: Static Audit +## The Test Methodology -Read the workflow files, `version.json`, and whatever else a check names as its own evidence: a project or dependency file, `global.json`, `codecov.yml`, `.gitignore`, the branch ruleset, and the repo's Actions and Dependabot secret names. Assert the structural fact behind each applicable D-guarantee, each pass, fail, or N/A with a `file:line` citation, cite a repository setting by its own name where that setting rather than a file is the evidence, and remember the two layers, asserting each input in the layer that declares it. `WORKFLOW.md` 5A carries the whole core list and the per-type addenda, and the sibling `d-guarantees.md` carries the guarantees each item answers to, so read this as an index into them rather than as the sweep itself. + -The core sweep reaches the paths-filter, naming each target's own build paths so a change touching none marks nothing. It reaches smoke gating on every upload. It reaches the aggregator's `needs:` and its skip and fail handling. It reaches coverage collection and its best-effort Codecov upload in every C# and Python repo that has tests at a profile other than `lint-only`, since a repo whose coverage never reaches Codecov passes every other check in this list. It reaches the entry validation jobs and the two-directional release gate. It reaches the single-branch NBGV classification, with the gate's default-branch literal, the `prerelease` expression, and `version.json`'s `publicReleaseRefSpec` all naming the repo's actual default branch. It reaches `target_commitish` from `GitCommitId`. It reaches the consume-then-delete artifact lifecycle, with `retention-days: 1` everywhere and no blanket delete. It reaches the `pattern:` handoff and `inputs.branch` config. It reaches the publisher's serialized concurrency and the SHA pins. Those are entry points into `WORKFLOW.md` 5A's core list rather than the whole of it. +An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a check or scenario for an absent construct is recorded N/A, not failed. -The per-type addenda cover .NET publish, NuGet, PyPI, Docker, and a static site deployed to a host, several assertions each. Apply only the ones the repo's types imply, and read them in `WORKFLOW.md` 5A rather than from this list. +### 5A. Static Audit (No Execution) -## 5B: Trace Scenarios +Read the workflow files, `version.json`, and whatever else a check below names as its evidence: a project or dependency file, `global.json`, `codecov.yml`, `.gitignore`, the branch ruleset, and the repo's Actions and Dependabot secret names. Assert the structural fact behind each *applicable* D-guarantee, each pass/fail/N-A with a `file:line` citation, and cite a repository setting by its own name where that setting rather than a file is the evidence. Remember the two layers: assert each input in the layer that declares it. -For each applicable scenario, evaluate every job's `if:`/`needs:` against the inputs and compare the predicted run/skip, version, release, and artifact end state to the expected table in `WORKFLOW.md` 5B. The load-bearing ones: +**Core (every repo):** -- **S1** a PR touching a target: that target smoke-builds, nothing uploads, the aggregator succeeds. -- **S5/S6** a bot push to `main`: publishes only when code-affecting, and a human push never does. -- **S7** a publish run builds the one trigger branch with the right classification and leaves no dangling artifacts. -- **S8** a dispatch from a ref other than `main`/`develop` fails fast. -- **S9** a no-op re-run on a schedule or push trigger: release-create skipped, registries dedupe, package build artifacts still deleted, Docker still re-pushes. A dispatch re-run refreshes the release instead. -- **S10** branch and version classification disagree: the gate fails loud and everything downstream skips. -- **S12/S13** a deploy dispatch: ref gate first, environment re-asserted, pointer flip separate, live check names the release, and a production deploy from a non-default ref fails before anything is written. +- **D1:** a `changes` paths-filter job exists wherever the repo has a smoke build, with one entry per target naming the paths that target is built from, so a change touching no target marks nothing (a filter written as a negation instead marks a docs-only change and fails D1.1); the PR entry workflow's smoke call sets every publish flag its release task declares to false (`github`/`dockerhub`, and a package-push flag there is itself a finding, per `WORKFLOW.md` section 1); a pushing leaf receives `smoke: true` and a derived `push` (false on smoke), and a build-only leaf receives `smoke: true` with no `push` to derive; every `upload-artifact` the smoke call reaches, in a build task and in any job collecting other jobs' artifacts alike, is gated on smoke being false, written `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings; the aggregator runs under `if: always()`, `needs:` the validation job, and the `changes` and `smoke-build` jobs too wherever the repo has a smoke build, blocks on `failure`/`cancelled`, and passes on a **skipped smoke build**, so a no-build repo's aggregator, having only the validation job to read, requires that job to have succeeded; the aggregator's own job `name:` is the string the branch ruleset's required-check `context:` carries; a validation job runs unconditionally. +- **D1.6:** the validator the repo's validation job reaches collects coverage and uploads it, in every C# and Python repo that has tests. Its C# leg runs `dotnet test --coverage --coverage-output-format cobertura --results-directory ./coverage`, with `--coverage-output` unset, wherever the repo's test project is MTP-based. An MTP-based repo also ships a root `global.json` declaring the Microsoft.Testing.Platform runner, references `Microsoft.Testing.Extensions.CodeCoverage` at 18.9.0 or later in place of `coverlet.collector`, and carries no `xunit.runner.visualstudio`, while a repo still on the VSTest collector keeps its existing validator pin, a migration owed rather than drift. Its Python leg runs `pytest --cov-report=xml`, and the repo declares `pytest-cov` among its test dependencies, a dev dependency group in a uv project and a `requirements*.txt` entry on pip, selects the coverage source in its own `pyproject.toml` rather than leaving `--cov` unset, and leaves the report at the repo root as `coverage.xml`. Either way the report reaches a `codecov/codecov-action` step made best-effort by `continue-on-error` and/or `fail_ci_if_error: false`, and `CODECOV_TOKEN` is present in **both** the repo's Actions and its Dependabot secret names, the second because a Dependabot-triggered run reads that store and the upload otherwise skips silently on every bot pull request. `codecov.yml` sets the project and patch statuses `informational: true`, or names the threshold the repo enforces instead, and lists any intentionally-untested, non-shipped project under `ignore`. `.gitignore` excludes the coverage output. Record the whole item N/A for a repo with no tests, and for a type the audited repo carries at the `lint-only` profile. +- **D2:** an entry validation job/step exists per complex-input workflow; the release gate checks both directions, strips `+buildmetadata`, and skips on smoke; the publisher rejects a dispatch from a ref other than `main` or `develop`. +- **D3:** each run builds one branch, so NBGV classifies `github.ref` directly (no `IGNORE_GITHUB_REF`), and the default-branch literal in the gate (`== 'main'`), the `prerelease` expression (`!= 'main'`), and `version.json`'s `publicReleaseRefSpec` all name the repo's actual default branch. `version.json` sets the major.minor floor, and NBGV and `version.json` are both retained even by a repo with no compiler, since they own the tag (D3.3). NuGet.org derives the prerelease flag from the SemVer2 `-g` suffix rather than the workflow setting one, and the PyPI version is built from `AssemblyFileVersion` with `.dev0` appended on `develop` only (D3.4). +- **D4:** `target_commitish` is the NBGV commit id; `prerelease` equals `branch != default`; the release-create step is gated `exists == 'false' || github.event_name == 'workflow_dispatch'` (the step output is the string `'false'`, not a boolean); the asset-delete step carries that same condition, narrowed by `inputs.expect_release_assets`. Where `workflow_dispatch` is the publisher's only trigger (`releaseTrigger: dispatch-only`) every run is a dispatch, so the exists-check's skip leg can never fire: record that leg N/A rather than failed, and expect the release-create step to still carry the `exists == 'false' || github.event_name == 'workflow_dispatch'` condition, since D6.4 keeps the `github-release` job body verbatim. A caller with no file target passes `expect_release_assets: false`, which covers a Docker-only, a PyPI-only, and a source-only repo alike, while a NuGet-only caller keeps the default `true` because its leaf uploads a `release-asset-*` carrying the package, and a source-only caller also sets every `enable_*` input false. `github-release` and the terminal registry pusher (Docker) each carry `!failure() && !cancelled()` rather than the implicit `success()`, so a failed build skips both, while a target that merely skipped, being disabled or unchanged, still lets the release and the Docker push proceed, and a package target's separate publish job `needs:` the release-task call for the same reason (D4.5). A first `plan` job decides once whether the run publishes, admitting a code-affecting bot push to `main`, a dispatch of `main` or `develop`, and a `main`-only schedule, and every publishing job gates on that decision (D4.1). +- **D5:** each cross-job transfer artifact has a delete step at its consumer, gated so it runs exactly when the consumption happened rather than when the whole job succeeded (D5.2), `continue-on-error: true`, tolerating a failed listing, and looping all ids; **every** upload sets `retention-days: 1`; **no** cleanup step enumerates and deletes the run's whole artifact set, in whatever jq or API shape it is written. +- **D6:** the release download uses `pattern:`/`merge-multiple:` (no `artifact-ids:`). Branch-derived config reads `inputs.branch` (a `github.ref_name` in such config is a finding). Artifact names are branch-suffixed. The target set is consistent across every surface D6.4 names: the `enable_` input, the `build-` job, that job's entries in the `github-release` and `build-docker` `needs:` lists, the `changes` paths-filter entry **and** its output, the `smoke-build` enable-forward, and any separate `publish-` job the package-registry seam requires. The `inputs.branch` rule above binds a called leaf, while a `publish-` job is in the publisher and reads `github.ref_name` correctly. +- **D7:** the publisher concurrency group is global and ref-independent with `cancel-in-progress: false`. A reusable job declares `permissions:` only where every caller grants that scope at startup, per D7.2. A boolean used by both `workflow_call` and `workflow_dispatch` is declared in both trigger blocks, and a boolean read through `github.event.inputs.` is compared against `'true'` alone, a comparison against the boolean `true` never firing on a string, while an `inputs.` read carries the declared boolean and is used directly, a both-forms comparison there being redundant rather than a finding, so a repo whose booleans all arrive by `workflow_call` records the comparison half N/A. A job or step output is a string too and takes the same `== 'true'` (D7.3). Every cross-job condition that admits a skipped dependency pairs its allowlist with a status-check function such as `always()` or `!failure() && !cancelled()`, since the implicit `success()` is false the moment a `needs:` job skipped (D7.4). +- **D8/D9:** the merge-bot enables auto-merge on `opened`/`reopened` for every Dependabot tier including semver-major, dispatches `--squash`/`--merge` by the PR's base ref, disables on a maintainer-pushed `synchronize`, and keys concurrency on the PR number rather than `github.ref`. Codegen runs as a matrix over both branches, and Dependabot targets both branches with security PRs to the default branch (D8.2). The upstream tracker's `bump-branch-prefix` and `branches` match a merge-bot rule, unless it sets `auto-merge: false`, which prefixes the head so no rule matches (wrapper repos). A gate comparing `github.actor` against hard-coded bot identities emits a `::warning::` on the non-matching branch, and the annotation is optional only where the failure announces itself anyway (D8.4). Actions are SHA-pinned. Names, shells, and conditionals follow `GOVERNANCE.md` "Workflow YAML Conventions", and line endings follow `.editorconfig` (D9.5), per the policy `GOVERNANCE.md` "Documentation Style Conventions" routes to under "Line Endings". -## 5C: Live Probe +**Per-type addenda (apply only the ones present):** -Only for what a static trace cannot settle. Every probe that dispatches a workflow, re-runs a real publish, or acts on the deploy host directly is the maintainer's to run: the agent prepares the command and reads the result back afterwards, and a harness refusal to fire one is the control working, never something to re-shape. The probes are a trivial PR to confirm S1, which runs same-repo only wherever the repo has a Docker leg, since that leg logs in to the registry even on smoke, registry queries after a real publish, the version classification and artifact lifecycle read from a real publish's logs, and the deploy ref gate, which is verified only by tripping it. That gate's evidence is four items, the gate job's conclusion, its error text naming the expected and the received ref, every downstream job recorded skipped rather than passed, and the production environment's deployment list carrying no deployment from the dispatched ref, because a gate that fails open and a gate nobody tripped leave the same empty run history behind. +- **.NET publish:** the smoke runtime set is a strict non-empty subset of the full runtime set. The selected set runs sequentially inside one composite-action job. A non-smoke run uploads one `release-asset--dotnet-publish` artifact, while a smoke run skips the archive and upload steps. +- **NuGet:** `publish-nuget` is a job in the repo's own publisher, never inside the release task and never in a reusable workflow a different repository hosts. `id-token: write` appears on that job only, absent from the build and PR paths, beside `actions: write` for the artifact cleanup. The push uses `--skip-duplicate` and is gated by that job's `needs:` on the release-task call, never on an existence check, so a PR never reaches it. The job consume-then-deletes `nuget-build-` under the download step's own success, per D5.1 and D5.2. `*.nupkg` push also carries the paired `.snupkg` to the symbol server where symbols are enabled. The `release-asset` `.7z` carries the package(s). +- **PyPI:** `publish-pypi` declares `environment: { name: pypi }`. `id-token: write` appears only on that job (absent from the build/PR path). `skip-existing: true` is set on the publish action. The build artifact is deleted under the download step's own success, per D5.2. The `pypi` environment has a deployment-branch rule. +- **Docker:** the leaf reads the external state file for the tag instead of `SemVer2` (wrapper repos only, since a plain Docker repo correctly tags off `SemVer2` and records this N/A). The readme job is gated main-only, both by the caller's branch input and inside the hub-hosted `publish-docker-readme-task.yml` itself. The docker-readme task validates `repositories` XOR `manifest`+`manifest-jq`. The buildcache follows D9.4. +- **Static site deployed to a host:** the generator is pinned by version **and** by a checksum verified before install, declared once across the workflows that install it. The deploy is a dispatch carrying an environment choice, with concurrency keyed on the **environment** and `cancel-in-progress: false`, and production gated to the default branch while any ref may reach a non-production environment. The reusable callee re-asserts the environment name in a job of its own. The upload targets a per-release directory and carries no delete flag at the environment root, and the pointer flip is a separate step. The terminal check asserts the environment, then the release id, then the URL contract, waiting for convergence to a bounded timeout rather than sampling once, and reporting an unreachable host distinctly from an HTTP status (D4.6). Retention is bounded by a declared count and one side is recorded as owning the prune: the deploy asserts it where the credential can observe the destination, and the host owns it where the credential is confined write-only (D5.6). -## Verdict +### 5B. End-to-End Trace Scenarios (No Execution, Deterministic from the YAML) -Record the workflow operational when every applicable 5A item passes, every applicable 5B scenario's predicted output equals the expected, and no 5C probe that was run contradicts either. Any applicable mismatch is a defect. The verdict names the failing guarantees with the triggering input for each, the items recorded N/A, and the 5C probes prepared but not run, so a static-only audit and a fully probed one do not read alike. Per-project-type walkthroughs mapping scenarios onto targets, including source-only, static-site, and operational shapes, are `WORKFLOW.md` section 6. +For each *applicable* scenario, evaluate every job's `if:`/`needs:` against the inputs and emit the predicted **run/skip + version + release + artifact-end-state** table, then compare to the expected. A scenario governing a construct the repo does not contain is N/A, per `WORKFLOW.md` section 1, and an absent trigger is such a construct: a dispatch-only publisher records S5, S6 and S9 N/A, since their push and schedule paths can never fire there. Minimum set: + +| # | Input | Expected output | Exercises | +| --- | --- | --- | --- | +| S1 | PR touching a build target | `changes` flags it; validation runs; that target's smoke build runs; no push, **no uploads**; validate-release **skipped (smoke), succeeds**; release **skipped**; aggregator **success**; version = prerelease; no release; no dangling artifacts | D1, D2.2, D3 | +| S2 | PR changing only docs | smoke-build **skipped**, validation runs, aggregator **success** | D1.1, D1.2, D1.5 | +| S3 | PR changing only `.github/workflows/**` | the filter marks no target -> smoke-build **skipped**, validation runs, aggregator **success** | D1.2, D1.4, D1.5 | +| S4 | PR base = default branch, carrying a build target | smoke versions as prerelease, validate-release **skipped (smoke), succeeds** so the default-branch arm does **not** fire, aggregator **success**, promotion not blocked | D1.5, D2.2, D3.2 | +| S5 | bot push to `main` not touching a release path (e.g. an Actions bump) | the paths filter excludes it, so nothing publishes | D4.1 | +| S6 | code-affecting **bot** push to `main` (a human push/promotion, or any develop push, does not) | the `plan` job gates it to the App/Dependabot actor, and `main` publishes a release | D3, D4 | +| S7 | publish run (schedule, a bot push to main, or a dispatch) | builds the **one** trigger branch: `main` -> `X.Y.Z`, `prerelease=false`, registry stable, readme run; `develop` -> `X.Y.Z-g`, `prerelease=true`, registry prerelease; `release-asset-*` consumed-then-deleted; each package build-artifact (`nuget-build-*`, `pypi-build-*`) deleted after its publish; **no dangling artifacts** | D3, D4, D5, D6, D7 | +| S8 | dispatch from a ref other than `main` or `develop` | **fails fast** | D2.3 | +| S9 | re-run publish on a schedule or push trigger, version unchanged (a dispatch re-run refreshes the release instead, per D4.4) | release-create **skipped**, `release-asset-*` delete **skipped**; NuGet/PyPI pushes no-op (server dedupe); **package build-artifacts still deleted** (their download succeeded); **Docker still re-pushes** the image; no duplicate release | D4.4, D5.2 | +| S10 | branch/version classification disagree | validate-release **fails loud**, build/publish skip | D2.2 | +| S11 | scheduled upstream-version bump (wrapper) | resolver detects a change -> commits the state file -> opens a `-` PR -> merge-bot auto-merges -> the `main` pin publishes via the gate (a develop pin does not auto-publish, shipping instead via a develop dispatch or promotion) | D8.3, D3.5 | +| S12 | deploy dispatch naming an environment | the ref gate runs **first** (production from the default branch only, any ref to a non-production environment); validation runs; the callee re-asserts the environment name; a release installs under its own id; the pointer flips as a separate step; retention is bounded by whichever of the two D5.6 shapes the repo uses, so a deploy whose credential can observe the destination asserts the count converged and one confined write-only leaves it to the host; the live check asserts the environment and the release id, waiting out the reload, then the URL contract; **no tag and no release are created** | D2.1, D4.6, D5.6 | +| S13 | deploy dispatch of a production environment from a non-default ref | **fails fast**, before anything is installed or written | D2.1 | + +### 5C. Live Probe (Where Warranted) + +Every probe here that dispatches a workflow or re-runs a real publish is the maintainer's to run, and the agent prepares the command and reads the result back afterwards. A harness that refuses such a write is the harness working as intended, and the refusal is neither re-shaped into a raw API call nor talked around (GOVERNANCE.md "Repository Boundaries and Write Safety"). + +- Open a trivial-change PR touching one target and confirm S1. *Caveat: the Docker leg logs in to the registry even on smoke and reads the buildcache, so it needs `DOCKER_HUB_*` secrets and cannot run on a fork PR (same-repo only).* +- Per registry: after a real publish, query NuGet.org for the expected version + prerelease classification (and the `.snupkg` on the symbol server), and confirm a re-run added no duplicate. For PyPI read the built `dist/*` filenames out of the build job's log, `.dev0` off `develop` vs a plain version on the default branch. +- Inspect the latest real publish's logs for `PublicRelease`/`SemVer2` per leg and confirm the artifact lifecycle (uploaded, consumed, deleted, with none left behind). +- **The deploy ref gate (S13) is verified only by tripping it.** Dispatch the production environment from a non-default ref and expect the run to fail at the gate. The evidence is four things, and each of them matters: the gate job's conclusion, its error text naming the expected and the received ref, every downstream job recorded as **skipped** rather than passed, and the production environment's deployment list carrying no deployment from the dispatched ref. Capture all four, because a gate that fails open and a gate nobody tripped produce the same empty run history, so "we have never seen it fail" is not evidence about the one control standing between a mis-dispatch and the live site. **The agent prepares the command and reads all four back afterwards. It does not fire it.** The same split applies to any probe that acts on the deploy host directly, an outbound SSH exercising a forced command among them. + +### Assessment + +Record the workflow **operational** when every *applicable* 5A item passes, every *applicable* 5B scenario's predicted output equals the expected, and no 5C probe that was run contradicts either. N/A items are excluded, never counted as failures. Any *applicable* mismatch is a **defect** -> **not operational**. Procedure: + +1. **Audit** with 5A. Record pass/fail/N-A with `file:line`. +2. **Trace** the applicable S-scenarios with 5B. Diff predicted vs expected. +3. **Probe** with 5C where a live signal exists that the static trace cannot produce: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. +4. **Verdict:** operational / not operational, with the failing guarantee(s) and the triggering input for each, the list of items recorded N/A, and the 5C probes prepared but not run. + +`WORKFLOW.md` section 5 keeps the test methodology, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/test-methodology.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. + + diff --git a/.github/skills/skill-lifecycle/SKILL.md b/.github/skills/skill-lifecycle/SKILL.md index 747a6c2fd..4a8aa6f89 100644 --- a/.github/skills/skill-lifecycle/SKILL.md +++ b/.github/skills/skill-lifecycle/SKILL.md @@ -46,7 +46,7 @@ A skill surfaces at a trigger moment. A rule that binds every action all the tim Packaging keeps one topic in one authoritative place while the skill makes it surface automatically. It has three shapes, and each pairing states which it uses: - **Moved content.** The law-doc section keeps a summary and the skill holds the full rules (`git-commit-conventions`, `comment-and-doc-style`, `pr-review-conduct`). The section ends with the standard pointer sentence: packaged as the named skill at `.agents/skills//SKILL.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, read the skill for the full rules. -- **Kept authority.** The source doc keeps the full rules and the skill is the summary that routes to them (`audit-a-repo` over `AUDIT.md`, `workflow-ci-contract` over `WORKFLOW.md` outside section 4). The skill states per topic which doc section owns it. -- **Included content.** The doc keeps the full rules and the skill needs them whole to work in isolation, so it carries the section as a generated include rather than as a summary or a copy, declared with the region markers "The Pipeline" above describes and keyed on the doc's section (`agent-conduct` over the three `GOVERNANCE.md` sections it surfaces, `workflow-ci-contract` over `WORKFLOW.md` section 4). The doc side states the shape with one sentence naming the skill that includes the section, and the skill side is the region itself. A section carried this way is read outside its own document, so it names a sibling section by document and heading rather than as above or below, and it links to no file by a relative path, since the path would resolve against the skill's directory rather than the doc's. The doc wins by construction, since `scripts/build_dist.py` writes the region from it and its `--check` reports a region that differs from it as stale. +- **Kept authority.** The source doc keeps the full rules and the skill is the summary that routes to them (`audit-a-repo` over `AUDIT.md`, `workflow-ci-contract` over `WORKFLOW.md` outside sections 3, 4, and 5). The skill states per topic which doc section owns it. +- **Included content.** The doc keeps the full rules and the skill needs them whole to work in isolation, so it carries the section as a generated include rather than as a summary or a copy, declared with the region markers "The Pipeline" above describes and keyed on the doc's section (`agent-conduct` over the three `GOVERNANCE.md` sections it surfaces, `workflow-ci-contract` over `WORKFLOW.md` sections 3, 4, and 5). The doc side states the shape with one sentence naming the skill that includes the section, and the skill side is the region itself. A section carried this way is read outside its own document, so it names a sibling section by document and heading rather than as above or below, and it links to no file by a relative path, since the path would resolve against the skill's directory rather than the doc's. The doc wins by construction, since `scripts/build_dist.py` writes the region from it and its `--check` reports a region that differs from it as stale. In every shape the doc is the authority when the two are found to disagree, the moved-content shape included: the doc's summary says what the rule is, and the skill's full text is what gets corrected. A deliberate change to a packaged rule is not such a disagreement. It lands where the full text lives, and in the same change the author either edits the other side's summary to match, since a summary has no mechanical check, or regenerates the include, which has one. A rule stated fully in both places by hand is the drift this pattern exists to prevent, and an include is the one full second statement that cannot drift undetected. diff --git a/.github/skills/workflow-ci-contract/SKILL.md b/.github/skills/workflow-ci-contract/SKILL.md index c761fb838..8f7514137 100644 --- a/.github/skills/workflow-ci-contract/SKILL.md +++ b/.github/skills/workflow-ci-contract/SKILL.md @@ -8,31 +8,23 @@ description: >- ## Why This Exists -`WORKFLOW.md` is the fleet's CI/CD behavioral contract. This skill is that contract's surface, so an agent editing workflow YAML has the contract in view. It carries the summary, with the contract's section 4 carried whole as a generated include and the test methodology indexed in `references/`. `WORKFLOW.md`'s own canonical-scope note says which of it and `GOVERNANCE.md` is authoritative where the two overlap. +`WORKFLOW.md` is the fleet's CI/CD behavioral contract. This skill is that contract's surface, so an agent editing workflow YAML has the contract in view. It carries the summary, and `WORKFLOW.md` sections 3, 4, and 5 are each carried whole in `references/` as a generated include. `WORKFLOW.md`'s own canonical-scope note says which of it and `GOVERNANCE.md` is authoritative where the two overlap. ## How the Contract Is Read - **Outcomes, not bytes.** A workflow is judged against `WORKFLOW.md` section 4's expected inputs and outputs, never against a snippet byte for byte, per `GOVERNANCE.md` "Foundational Principles". -- **Applicability.** A guarantee governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. +- **Applicability.** A guarantee, or a 5A check or 5B scenario from `WORKFLOW.md` section 5, governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. - **Operational is binary.** Every applicable guarantee holds, or the workflow is not operational. A single applicable input-output mismatch is a defect regardless of how clean the YAML looks. -- **Reached, not carried.** A standard workflow whose job graph is identical across repos of a type is reached as a hub-hosted `workflow_call` task, per `GOVERNANCE.md` "Hub-Hosted Tooling". The repo's own surface is the caller stub, pinned to a hub release commit, and a composite-action hook at `.github/actions/` for what is its own. A hub task reaches its own actions and sibling tasks through `$/`, which resolves at that pinned commit. The merge-bot is the first, and `docs/reusable-workflows.md` in the hub carries the model, the hook contract, and the phase each workflow migrates in. Until a workflow's phase ships, its copy is graded as below. -- **Two layers.** Orchestration (the PR entry workflow, publisher, version and release jobs) is generic and standard at the job level. Build leaves (the `build-` tasks) are repo-owned. Inputs like `github`/`dockerhub`/`expect_release_assets` live on the orchestrator, a leaf receives `ref`/`branch`/`smoke` and whatever else its target needs, a derived `push` among them where that leaf pushes, so assert each input in the layer that declares it. A package target declares no push input on either layer, its push living in a separate `publish-` job in the repo's own publisher. What a repo curates is the list of targets, and adding or dropping one edits the whole surface together: the `enable_` input, the `build-` job and its `github-release` and `build-docker` `needs:` entries, the `changes` paths-filter entry and output, the `smoke-build` enable-forward, and a package target's `publish-` job (D6.4). +- **Reached, not carried.** A standard workflow whose job graph is identical across repos of a type is reached as a hub-hosted `workflow_call` task, per `GOVERNANCE.md` "Hub-Hosted Tooling". The repo's own surface is the caller stub, pinned to a hub release commit, and a composite-action hook at `.github/actions/` for what is its own. A hub task reaches its own actions and sibling tasks through `$/`, which resolves at that pinned commit. The merge-bot is the first, and `docs/reusable-workflows.md` in the hub carries the model, the hook contract, and the stage each workflow migrates in. Until a workflow's stage ships, its copy is graded against the same contract. +- **Two layers.** The pipeline splits into an orchestrator layer and a build-leaf layer, defined in `WORKFLOW.md` section 3's `Two Layers: Orchestration vs Build` and carried in `references/architecture.md`, while `WORKFLOW.md` section 1's `Two layers when auditing` maps which layer declares which input. When a check names an input, assert it in the layer that declares it. ## Style Rules `GOVERNANCE.md` "Workflow YAML Conventions" keeps the style rules, and the `comment-and-doc-style` Skill keeps the line-ending policy, reached from `GOVERNANCE.md` "Documentation Style Conventions" under "Line Endings". Read both before editing a workflow or a composite action. -## The Core Behavioral Spine +## The Contract Text -- **PRs validate fast and never publish**: a paths-filter smoke-builds only changed targets, the caller's own job reaching the reusable validator, or the replacement it points its aggregator at, always runs, and one required aggregator gates the merge, running under `if: always()` so a failed or skipped dependency cannot skip the gate itself, treating skipped smoke as pass and blocking on failure or cancelled. Smoke does a full compile/lint/test but pushes nothing and uploads nothing, every `upload-artifact` gated on smoke being false, which is `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings. -- **A human merge never auto-publishes**, per `GOVERNANCE.md` "Release Model": a `plan` job decides once and every job gates on it. Publishes come from a code-affecting bot push to `main`, a manual dispatch of `main` or `develop`, or the main-only weekly Docker schedule, while a publisher whose only trigger is `workflow_dispatch` (`releaseTrigger: dispatch-only`) reaches the dispatch alone, its bot-push and schedule paths never firing, which covers a source-only repo and an operational repo alike. Each run builds the one trigger branch, the default branch a clean `X.Y.Z`, anything else a prerelease `X.Y.Z-g`, with NBGV owning the patch from git height. The gate's branch literal, the `prerelease` expression, and `version.json`'s `publicReleaseRefSpec` each name the repo's actual default branch, and a divergence among the three is a defect. The release tags the built commit's SHA (`GitCommitId`), never a branch name. -- **Validate at entry**: `GOVERNANCE.md` "Workflow YAML Conventions" keeps the rule. The release gate that applies it checks branch-versus-prerelease in both directions, strips `+buildmetadata`, and on smoke skips the check while the job still succeeds. -- **The seam contract**: a target contributes a release file by uploading `release-asset--`, and the release job collects by `pattern:` plus `merge-multiple:`, never `artifact-ids:`, canonical even for a single target. A caller with no file target passes `expect_release_assets: false`, which covers a Docker-only, a PyPI-only, and a source-only repo, while a NuGet-only caller keeps the default `true`, its leaf uploading a `release-asset-*` that carries the package. -- **Artifacts are an intra-run handoff**: `GOVERNANCE.md` "Workflow YAML Conventions" keeps the consume-then-delete rule, the condition that gates the delete, and the blanket-delete ban. D5 in `references/d-guarantees.md` states the outputs a pipeline shows for them. -- **No-op republish**: an unchanged version re-pushes nothing, the release-create step skips when the tag exists and is refreshed only on a dispatch, registries dedupe server-side (`--skip-duplicate`, `skip-existing: true`), and Docker alone always re-pushes by design. -- **A build failure blocks every publish target**: `github-release` needs every build and guards with `!failure() && !cancelled()` as the terminal registry pusher (Docker) does, and a package target's separate `publish-` job `needs:` the release-task call, so no build failure ships anything partial. A failed package push is outside that, since it runs after the release is cut. - -`references/d-guarantees.md` carries `WORKFLOW.md` section 4 whole as a generated include, so a guarantee's exact wording is one read away. `references/test-methodology.md` indexes `WORKFLOW.md` section 5's audit, trace, and probe procedure, and the sweep itself is run from section 5, which carries the whole core list, the per-type addenda, and the scenario table. +`references/architecture.md`, `references/d-guarantees.md`, and `references/test-methodology.md` carry `WORKFLOW.md` sections 3, 4, and 5 whole, each as a generated include, so the pipeline's architecture, a guarantee's exact wording, and the audit-trace-probe procedure are each one read away rather than restated in full here. A defect in an include region is fixed in `WORKFLOW.md` and regenerated, never edited in this skill, per the `skill-lifecycle` Skill. `WORKFLOW.md` keeps sections 1, 2, and 6 itself, the applicability rule, the style-rule pointer, and the per-project-type walkthroughs that say which items go N/A per type, so read those there. ## After Any Workflow Edit diff --git a/.github/skills/workflow-ci-contract/references/architecture.md b/.github/skills/workflow-ci-contract/references/architecture.md new file mode 100644 index 000000000..cfd6598a2 --- /dev/null +++ b/.github/skills/workflow-ci-contract/references/architecture.md @@ -0,0 +1,112 @@ +# The Pipeline Architecture + +The section below is `WORKFLOW.md` section 3, whole. The D-guarantees it cites by number are `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. + +## The Architecture + + + +### Branch Model + +Two workflow models, set per repo by the registry `workflowModel` field. `release` (default) is the feature-branch pipeline `WORKFLOW.md` specifies: + +```mermaid +flowchart LR + feature[feature branch] -->|squash| develop + develop -->|merge commit| main + main -.->|no back-merge| develop +``` + +`operational` repos (live-service config, `workflowModel: operational`) commit directly to `develop` and promote a known-good snapshot to `main` via an occasional PR: + +```mermaid +flowchart LR + edit[direct signed commit] -->|advisory CI| develop + pr[pull request] -->|lint CI, reported not required| develop + develop -->|merge commit, enforced lint CI| main +``` + +The direct commit is an **allowance, not a substitute for review**. The ruleset drops the pull-request *requirement*, which permits a direct push without withdrawing the pull request, so a change worth reviewing still takes one and both paths reach `develop` legally. Which changes those are is stated as a shape rather than a line count in `GOVERNANCE.md` "Operational Repositories", which owns the test and is the one place it is written, since nothing in a ruleset can apply it. What differs is when validation lands. On the direct-commit path the commit is already on the branch, so CI can only be advisory after the fact, and that is the accepted cost of the model. On the pull-request path the change has not landed, so validation is pre-merge and actionable, which is the moment it is worth the most, and the lint workflow's `pull_request` trigger therefore names `develop` alongside `main` (`WORKFLOW.md` section 6). That is what makes **D1.2** hold here, since its input is *any* PR and the operational model is no exception. The check is reported on a `develop` PR rather than required, because a required status check on `develop` binds the direct push too and would dissolve the allowance the model is built on. + +Their CI is lint/validation only (editorconfig/EOL plus domain linters such as Home Assistant or ESPHome config validation or a firmware build, but **no unit tests**), so the D-guarantees in `WORKFLOW.md` section 4 that assume a build/test pipeline are **N/A** exactly as for `source-only` (`WORKFLOW.md` section 6). What binds: the promotion gate, where the `develop -> main` PR must pass the required `Check pull request workflow status job`, and the source-only release on manual dispatch (`releaseTrigger: dispatch-only`; tag + source zip). Branch-model rulesets are specified in `GOVERNANCE.md` "Branching Model" rather than in `WORKFLOW.md`. + +### Two Layers: Orchestration vs Build + +- **Orchestration** is generic and forms the standardization baseline **at the job level**: the single-branch publisher, the `get-version`, `validate-release`, and `github-release` jobs, and the `changes -> smoke-build -> aggregator` shape of the PR workflow. These job *bodies* should not need per-repo edits. +- **Build** is repo-owned in shape: the `build-` leaf tasks, whether this repo hosts them itself or reaches hub-hosted ones by pin. +- **What the repo curates** (by design, not a leak): the *list* of targets. This is **not** a byte-for-byte file carry. Adding or dropping a target edits the orchestrator's surface: the `enable_` inputs and the `build-` job + its `github-release` **and** `build-docker` `needs:` entries in the release task, **and** the `changes` paths-filter entry + output + the `smoke-build` enable-forward in the PR workflow, plus the separate `publish-` job for a package target. "Verbatim" applies to the `github-release` job and the version/publish-plan logic, not to the release task's job list or the paths-filter. Subsetting is symmetric: the same surface you trim to drop a target you extend to add a new one (e.g. a `release-asset--library` producer needs a new `enable_library` input, a `build-library` job, its two `needs:` entries, and a `library` paths-filter entry, output, and `smoke-build` enable-forward). + +### The Seam Contract + +A target contributes a file to the GitHub release by uploading a workflow artifact named `release-asset--`. The release job collects **every** matching artifact by **pattern** (`pattern: release-asset--*` + `merge-multiple: true`), never an `artifact-ids:` naming one job's output. Canonical for **every** repo, single-target included. Switching to an `artifact-id` handoff forks the release download and breaks the verbatim carry. + +```mermaid +flowchart LR + dotnet[dotnet-publish] -->|release-asset-BRANCH-dotnet-publish| store[(run artifacts)] + nuget[build-nuget] -->|release-asset-BRANCH-nuget| store + store -->|pattern + merge-multiple| rel["github-release job (D6)"] + nuget -->|nuget-build-BRANCH| pub["publish-TARGET job in the repo's own publisher"] + pypi[build-pypi] -->|pypi-build-BRANCH| pub + pub -->|push| registries[(registries)] + docker[build-docker] -->|push| registries +``` + +The diagram writes `BRANCH` and `TARGET` where the prose writes `` and ``, because a mermaid label is sanitized as HTML at render and an angle-bracket placeholder is dropped as an unknown tag. This reaches node labels as well as edge labels, which is why the Release Model diagram below writes `X.Y.Z-g-sha` rather than bracketing its own placeholder. + +### Reusable-Task Parameter Contract + +Every leaf and the release task take `ref`, `branch` (the **logical** branch that drives config/tags/prerelease), and where relevant `smoke`. Branch-derived config keys off `inputs.branch` (the logical branch the caller passes). Artifact names are branch-suffixed. + +### Versioning + +NBGV versions the branch being published. Each run builds a single branch (the trigger ref), so `GITHUB_REF` already names it and NBGV classifies it directly, and no `IGNORE_GITHUB_REF` override is required. The default branch is the public-release ref, so it builds clean `X.Y.Z`. Every other branch builds a prerelease `X.Y.Z-g`. `version.json`'s `version` is the major.minor floor. NBGV appends the git height as the patch. **NBGV and `version.json` are retained even by a repo with no compiled code**, since they are the source of the release tag (`SemVer2`) and `target_commitish` (`GitCommitId`) and the prerelease classification. The .NET SDK is pulled in only as the versioning toolchain. A package build derives its registry version from the same NBGV outputs, but **not always from `SemVer2`**: the PyPI version is built from `AssemblyFileVersion` (four-part `M.N.P.B`) with a PEP 440 `.dev0` appended on the `develop` branch. A wrapper repo may drive its build/image version from an external committed `name -> version` state file while NBGV still tags the release. + +### Validate-at-Entry + +When a workflow's inputs carry a cross-input or input-versus-derived-state invariant, assert it **once** in a dedicated entry job/step the downstream jobs `needs:`, failing fast with `::error::` before any build or publish. + +### Resource Lifecycle + +Workflow artifacts are an **intra-run handoff** only. Durable copies live on the release/registry. The rule: a transfer artifact handed **between jobs** is deleted by exact name/pattern **at its point of consumption**, the delete is **gated to the half of the consumption whose failure would leave it not yet redundant** (D5.2 names the two halves), and it is **best-effort**. **Every** `upload-artifact` sets `retention-days: 1` as the universal failure-path backstop, so no terminal blanket-delete job is needed. An intermediate consumed only within the same run may rely on the retention backstop alone. The run is **never** blanket-deleted (`.artifacts[].id`). See D5. + +### Fast PR Feedback + +PRs validate fast and never publish: a paths-filter smoke-builds only changed targets. A validation job always runs. Smoke builds compile/lint/test but upload nothing and push nothing. One required aggregator gates the merge. See D1. + +```mermaid +flowchart TD + pr[pull request] --> ch[changes paths-filter] + ch -->|target changed| sb[smoke-build changed targets] + ch -->|workflow-only or docs| skip[smoke-build skipped] + val[validation job] --> agg["Check pull request workflow status job (D1)"] + sb --> agg + skip --> agg + agg -->|success| ok[merge allowed] +``` + +### Release Model + +Each publish builds a **single branch**, the trigger ref (`main` a release, `develop` a prerelease), so there is no branch matrix and `github.ref` always names the built branch. A **human merge never auto-publishes**: a first `plan` job (`publish-plan-task.yml`) decides once and every job gates on it. A run publishes on a **code-affecting bot push to `main`** (the App merges every Dependabot/codegen PR, so `github.actor` gates it, and a shared paths filter also drops a non-substantive change like an Actions bump), a **manual dispatch** of `main`/`develop`, or a **main-only weekly schedule** (Docker, to refresh the base image). The `push` is main-only, so a develop bot merge publishes nothing (its prerelease comes via dispatch). A **source-only** repo publishes on **dispatch only**. Every release is a tag on the built commit plus a source archive, README, and LICENSE. Targets amend it with `release-asset-*` files, and a registry push contributes none, made by the Docker leaf for an image and by the separate `publish-` job for a package. An unchanged version re-pushes nothing (no-op republish). Docker re-pushes by design. + +```mermaid +flowchart TD + trig[main-only schedule / dispatch / paths-filtered push] --> one[build the one trigger branch] + one -->|main| vmain["version X.Y.Z stable (D3)"] + one -->|develop| vdev["version X.Y.Z-g-sha prerelease (D3)"] + vmain --> relm["github-release + registries: latest (D4)"] + vdev --> reld["github-release + registries: prerelease (D4)"] +``` + +### Output Seam by Destination + +Pick each output's path by **where the artifact goes**: + +- **File on the GitHub release** (zip, binary, packaged library): one leaf per output uploading `release-asset--`. The repo keeps `expect_release_assets: true` (its default). +- **Package-registry push** (NuGet, PyPI): the leaf builds and uploads a build artifact (`nuget-build-` / `pypi-build-`), and a separate `publish-` job in the **publishing repository's own** publisher consumes it and pushes. Both registries publish through OIDC Trusted Publishing, never a stored API key, and two things put that push outside the leaf. Trusted publishing validates the OIDC token's `job_workflow_ref` claim, which names the workflow the job actually ran from, so a push made from a reusable workflow a *different* repository hosts is rejected at the token exchange, NuGet.org answering `HTTP 401` with `does not start with //.github/workflows/`. That alone rules out a leaf another repository hosts. A leaf this repository hosts clears the claim, and the split still applies to it, because a called job declaring no `permissions:` runs under the calling job's whole grant, so a push anywhere inside the release task would put `id-token: write` on every job in it rather than at the one entry point D7.2 requires. The registered trusted-publishing policy therefore names the publisher, `publish-release.yml`. PyPI additionally gates its publish job behind an environment. NuGet.org binds its policy to the workflow file rather than to an environment and needs none. NuGet's leaf also uploads a `release-asset-*` carrying the package, and PyPI contributes none. +- **Image-registry push** (Docker): the leaf pushes the default branch multi-arch (amd64+arm64) and any other branch `amd64`-only (arm64 emulation is reserved for the released image), and contributes no `release-asset-*`. +- **Filesystem on a host the project owns** (a static site, a config tree): the leaf builds the tree, ships it to the host, and contributes no `release-asset-*`. The transport is the repo's own. What the contract fixes is that the deploy is a **separate `workflow_dispatch`** from the release, so a redeploy of an unchanged commit mints no tag and a host rebuild, a rollback, or proving a branch on a non-production environment costs nothing; that its credentials come from a **per-environment GitHub Environment** rather than the repository secret store; and that the deploy ends by asserting **what the host serves** rather than the transport's exit status (D4.6). Retention at the destination is bounded by a declared count with one side recorded as owning the prune, which is the deploy where its credential can observe the destination and the host where that credential is deliberately write-only (D5.6). +- **No file target via the release task** (Docker-only, PyPI-only, source-only): the release is tag + source zip + README + LICENSE. The caller **MUST pass `expect_release_assets: false`** to the release task. A publisher with file targets retains the default `true`. This setting is caller-specific. The default `true` fails on `fail_on_unmatched_files` when no assets exist. A **source-only** repo also passes every `enable_*` input as false because it has no build leaf (see `WORKFLOW.md` section 6). + +`WORKFLOW.md` section 3 keeps the architecture, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/architecture.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. + + diff --git a/.github/skills/workflow-ci-contract/references/d-guarantees.md b/.github/skills/workflow-ci-contract/references/d-guarantees.md index f33376a97..f2c626102 100644 --- a/.github/skills/workflow-ci-contract/references/d-guarantees.md +++ b/.github/skills/workflow-ci-contract/references/d-guarantees.md @@ -1,6 +1,6 @@ # The D-Guarantees -The section below is `WORKFLOW.md` section 4, whole. Which of its items bind a given repository is `WORKFLOW.md` section 1's applicability rule. +The section below is `WORKFLOW.md` section 4, whole. Which of its items bind a given repository is `WORKFLOW.md` section 1's applicability rule. The architecture these items govern is `WORKFLOW.md` section 3 and the methodology that checks them is `WORKFLOW.md` section 5, carried whole in `architecture.md` and `test-methodology.md` beside this file. ## The Behavioral Contract diff --git a/.github/skills/workflow-ci-contract/references/test-methodology.md b/.github/skills/workflow-ci-contract/references/test-methodology.md index d1e443cf3..744bff09f 100644 --- a/.github/skills/workflow-ci-contract/references/test-methodology.md +++ b/.github/skills/workflow-ci-contract/references/test-methodology.md @@ -1,31 +1,75 @@ # Testing a Repo's Workflows -The three escalating verification modes from `WORKFLOW.md` section 5, which keeps authority. N/A items (a check or scenario for an absent construct) are recorded and excluded, never failed. +The section below is `WORKFLOW.md` section 5, whole. Which of its checks and scenarios bind a given repository is `WORKFLOW.md` section 1's applicability rule, and the D-guarantees each one answers to are `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. -## 5A: Static Audit +## The Test Methodology -Read the workflow files, `version.json`, and whatever else a check names as its own evidence: a project or dependency file, `global.json`, `codecov.yml`, `.gitignore`, the branch ruleset, and the repo's Actions and Dependabot secret names. Assert the structural fact behind each applicable D-guarantee, each pass, fail, or N/A with a `file:line` citation, cite a repository setting by its own name where that setting rather than a file is the evidence, and remember the two layers, asserting each input in the layer that declares it. `WORKFLOW.md` 5A carries the whole core list and the per-type addenda, and the sibling `d-guarantees.md` carries the guarantees each item answers to, so read this as an index into them rather than as the sweep itself. + -The core sweep reaches the paths-filter, naming each target's own build paths so a change touching none marks nothing. It reaches smoke gating on every upload. It reaches the aggregator's `needs:` and its skip and fail handling. It reaches coverage collection and its best-effort Codecov upload in every C# and Python repo that has tests at a profile other than `lint-only`, since a repo whose coverage never reaches Codecov passes every other check in this list. It reaches the entry validation jobs and the two-directional release gate. It reaches the single-branch NBGV classification, with the gate's default-branch literal, the `prerelease` expression, and `version.json`'s `publicReleaseRefSpec` all naming the repo's actual default branch. It reaches `target_commitish` from `GitCommitId`. It reaches the consume-then-delete artifact lifecycle, with `retention-days: 1` everywhere and no blanket delete. It reaches the `pattern:` handoff and `inputs.branch` config. It reaches the publisher's serialized concurrency and the SHA pins. Those are entry points into `WORKFLOW.md` 5A's core list rather than the whole of it. +An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a check or scenario for an absent construct is recorded N/A, not failed. -The per-type addenda cover .NET publish, NuGet, PyPI, Docker, and a static site deployed to a host, several assertions each. Apply only the ones the repo's types imply, and read them in `WORKFLOW.md` 5A rather than from this list. +### 5A. Static Audit (No Execution) -## 5B: Trace Scenarios +Read the workflow files, `version.json`, and whatever else a check below names as its evidence: a project or dependency file, `global.json`, `codecov.yml`, `.gitignore`, the branch ruleset, and the repo's Actions and Dependabot secret names. Assert the structural fact behind each *applicable* D-guarantee, each pass/fail/N-A with a `file:line` citation, and cite a repository setting by its own name where that setting rather than a file is the evidence. Remember the two layers: assert each input in the layer that declares it. -For each applicable scenario, evaluate every job's `if:`/`needs:` against the inputs and compare the predicted run/skip, version, release, and artifact end state to the expected table in `WORKFLOW.md` 5B. The load-bearing ones: +**Core (every repo):** -- **S1** a PR touching a target: that target smoke-builds, nothing uploads, the aggregator succeeds. -- **S5/S6** a bot push to `main`: publishes only when code-affecting, and a human push never does. -- **S7** a publish run builds the one trigger branch with the right classification and leaves no dangling artifacts. -- **S8** a dispatch from a ref other than `main`/`develop` fails fast. -- **S9** a no-op re-run on a schedule or push trigger: release-create skipped, registries dedupe, package build artifacts still deleted, Docker still re-pushes. A dispatch re-run refreshes the release instead. -- **S10** branch and version classification disagree: the gate fails loud and everything downstream skips. -- **S12/S13** a deploy dispatch: ref gate first, environment re-asserted, pointer flip separate, live check names the release, and a production deploy from a non-default ref fails before anything is written. +- **D1:** a `changes` paths-filter job exists wherever the repo has a smoke build, with one entry per target naming the paths that target is built from, so a change touching no target marks nothing (a filter written as a negation instead marks a docs-only change and fails D1.1); the PR entry workflow's smoke call sets every publish flag its release task declares to false (`github`/`dockerhub`, and a package-push flag there is itself a finding, per `WORKFLOW.md` section 1); a pushing leaf receives `smoke: true` and a derived `push` (false on smoke), and a build-only leaf receives `smoke: true` with no `push` to derive; every `upload-artifact` the smoke call reaches, in a build task and in any job collecting other jobs' artifacts alike, is gated on smoke being false, written `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings; the aggregator runs under `if: always()`, `needs:` the validation job, and the `changes` and `smoke-build` jobs too wherever the repo has a smoke build, blocks on `failure`/`cancelled`, and passes on a **skipped smoke build**, so a no-build repo's aggregator, having only the validation job to read, requires that job to have succeeded; the aggregator's own job `name:` is the string the branch ruleset's required-check `context:` carries; a validation job runs unconditionally. +- **D1.6:** the validator the repo's validation job reaches collects coverage and uploads it, in every C# and Python repo that has tests. Its C# leg runs `dotnet test --coverage --coverage-output-format cobertura --results-directory ./coverage`, with `--coverage-output` unset, wherever the repo's test project is MTP-based. An MTP-based repo also ships a root `global.json` declaring the Microsoft.Testing.Platform runner, references `Microsoft.Testing.Extensions.CodeCoverage` at 18.9.0 or later in place of `coverlet.collector`, and carries no `xunit.runner.visualstudio`, while a repo still on the VSTest collector keeps its existing validator pin, a migration owed rather than drift. Its Python leg runs `pytest --cov-report=xml`, and the repo declares `pytest-cov` among its test dependencies, a dev dependency group in a uv project and a `requirements*.txt` entry on pip, selects the coverage source in its own `pyproject.toml` rather than leaving `--cov` unset, and leaves the report at the repo root as `coverage.xml`. Either way the report reaches a `codecov/codecov-action` step made best-effort by `continue-on-error` and/or `fail_ci_if_error: false`, and `CODECOV_TOKEN` is present in **both** the repo's Actions and its Dependabot secret names, the second because a Dependabot-triggered run reads that store and the upload otherwise skips silently on every bot pull request. `codecov.yml` sets the project and patch statuses `informational: true`, or names the threshold the repo enforces instead, and lists any intentionally-untested, non-shipped project under `ignore`. `.gitignore` excludes the coverage output. Record the whole item N/A for a repo with no tests, and for a type the audited repo carries at the `lint-only` profile. +- **D2:** an entry validation job/step exists per complex-input workflow; the release gate checks both directions, strips `+buildmetadata`, and skips on smoke; the publisher rejects a dispatch from a ref other than `main` or `develop`. +- **D3:** each run builds one branch, so NBGV classifies `github.ref` directly (no `IGNORE_GITHUB_REF`), and the default-branch literal in the gate (`== 'main'`), the `prerelease` expression (`!= 'main'`), and `version.json`'s `publicReleaseRefSpec` all name the repo's actual default branch. `version.json` sets the major.minor floor, and NBGV and `version.json` are both retained even by a repo with no compiler, since they own the tag (D3.3). NuGet.org derives the prerelease flag from the SemVer2 `-g` suffix rather than the workflow setting one, and the PyPI version is built from `AssemblyFileVersion` with `.dev0` appended on `develop` only (D3.4). +- **D4:** `target_commitish` is the NBGV commit id; `prerelease` equals `branch != default`; the release-create step is gated `exists == 'false' || github.event_name == 'workflow_dispatch'` (the step output is the string `'false'`, not a boolean); the asset-delete step carries that same condition, narrowed by `inputs.expect_release_assets`. Where `workflow_dispatch` is the publisher's only trigger (`releaseTrigger: dispatch-only`) every run is a dispatch, so the exists-check's skip leg can never fire: record that leg N/A rather than failed, and expect the release-create step to still carry the `exists == 'false' || github.event_name == 'workflow_dispatch'` condition, since D6.4 keeps the `github-release` job body verbatim. A caller with no file target passes `expect_release_assets: false`, which covers a Docker-only, a PyPI-only, and a source-only repo alike, while a NuGet-only caller keeps the default `true` because its leaf uploads a `release-asset-*` carrying the package, and a source-only caller also sets every `enable_*` input false. `github-release` and the terminal registry pusher (Docker) each carry `!failure() && !cancelled()` rather than the implicit `success()`, so a failed build skips both, while a target that merely skipped, being disabled or unchanged, still lets the release and the Docker push proceed, and a package target's separate publish job `needs:` the release-task call for the same reason (D4.5). A first `plan` job decides once whether the run publishes, admitting a code-affecting bot push to `main`, a dispatch of `main` or `develop`, and a `main`-only schedule, and every publishing job gates on that decision (D4.1). +- **D5:** each cross-job transfer artifact has a delete step at its consumer, gated so it runs exactly when the consumption happened rather than when the whole job succeeded (D5.2), `continue-on-error: true`, tolerating a failed listing, and looping all ids; **every** upload sets `retention-days: 1`; **no** cleanup step enumerates and deletes the run's whole artifact set, in whatever jq or API shape it is written. +- **D6:** the release download uses `pattern:`/`merge-multiple:` (no `artifact-ids:`). Branch-derived config reads `inputs.branch` (a `github.ref_name` in such config is a finding). Artifact names are branch-suffixed. The target set is consistent across every surface D6.4 names: the `enable_` input, the `build-` job, that job's entries in the `github-release` and `build-docker` `needs:` lists, the `changes` paths-filter entry **and** its output, the `smoke-build` enable-forward, and any separate `publish-` job the package-registry seam requires. The `inputs.branch` rule above binds a called leaf, while a `publish-` job is in the publisher and reads `github.ref_name` correctly. +- **D7:** the publisher concurrency group is global and ref-independent with `cancel-in-progress: false`. A reusable job declares `permissions:` only where every caller grants that scope at startup, per D7.2. A boolean used by both `workflow_call` and `workflow_dispatch` is declared in both trigger blocks, and a boolean read through `github.event.inputs.` is compared against `'true'` alone, a comparison against the boolean `true` never firing on a string, while an `inputs.` read carries the declared boolean and is used directly, a both-forms comparison there being redundant rather than a finding, so a repo whose booleans all arrive by `workflow_call` records the comparison half N/A. A job or step output is a string too and takes the same `== 'true'` (D7.3). Every cross-job condition that admits a skipped dependency pairs its allowlist with a status-check function such as `always()` or `!failure() && !cancelled()`, since the implicit `success()` is false the moment a `needs:` job skipped (D7.4). +- **D8/D9:** the merge-bot enables auto-merge on `opened`/`reopened` for every Dependabot tier including semver-major, dispatches `--squash`/`--merge` by the PR's base ref, disables on a maintainer-pushed `synchronize`, and keys concurrency on the PR number rather than `github.ref`. Codegen runs as a matrix over both branches, and Dependabot targets both branches with security PRs to the default branch (D8.2). The upstream tracker's `bump-branch-prefix` and `branches` match a merge-bot rule, unless it sets `auto-merge: false`, which prefixes the head so no rule matches (wrapper repos). A gate comparing `github.actor` against hard-coded bot identities emits a `::warning::` on the non-matching branch, and the annotation is optional only where the failure announces itself anyway (D8.4). Actions are SHA-pinned. Names, shells, and conditionals follow `GOVERNANCE.md` "Workflow YAML Conventions", and line endings follow `.editorconfig` (D9.5), per the policy `GOVERNANCE.md` "Documentation Style Conventions" routes to under "Line Endings". -## 5C: Live Probe +**Per-type addenda (apply only the ones present):** -Only for what a static trace cannot settle. Every probe that dispatches a workflow, re-runs a real publish, or acts on the deploy host directly is the maintainer's to run: the agent prepares the command and reads the result back afterwards, and a harness refusal to fire one is the control working, never something to re-shape. The probes are a trivial PR to confirm S1, which runs same-repo only wherever the repo has a Docker leg, since that leg logs in to the registry even on smoke, registry queries after a real publish, the version classification and artifact lifecycle read from a real publish's logs, and the deploy ref gate, which is verified only by tripping it. That gate's evidence is four items, the gate job's conclusion, its error text naming the expected and the received ref, every downstream job recorded skipped rather than passed, and the production environment's deployment list carrying no deployment from the dispatched ref, because a gate that fails open and a gate nobody tripped leave the same empty run history behind. +- **.NET publish:** the smoke runtime set is a strict non-empty subset of the full runtime set. The selected set runs sequentially inside one composite-action job. A non-smoke run uploads one `release-asset--dotnet-publish` artifact, while a smoke run skips the archive and upload steps. +- **NuGet:** `publish-nuget` is a job in the repo's own publisher, never inside the release task and never in a reusable workflow a different repository hosts. `id-token: write` appears on that job only, absent from the build and PR paths, beside `actions: write` for the artifact cleanup. The push uses `--skip-duplicate` and is gated by that job's `needs:` on the release-task call, never on an existence check, so a PR never reaches it. The job consume-then-deletes `nuget-build-` under the download step's own success, per D5.1 and D5.2. `*.nupkg` push also carries the paired `.snupkg` to the symbol server where symbols are enabled. The `release-asset` `.7z` carries the package(s). +- **PyPI:** `publish-pypi` declares `environment: { name: pypi }`. `id-token: write` appears only on that job (absent from the build/PR path). `skip-existing: true` is set on the publish action. The build artifact is deleted under the download step's own success, per D5.2. The `pypi` environment has a deployment-branch rule. +- **Docker:** the leaf reads the external state file for the tag instead of `SemVer2` (wrapper repos only, since a plain Docker repo correctly tags off `SemVer2` and records this N/A). The readme job is gated main-only, both by the caller's branch input and inside the hub-hosted `publish-docker-readme-task.yml` itself. The docker-readme task validates `repositories` XOR `manifest`+`manifest-jq`. The buildcache follows D9.4. +- **Static site deployed to a host:** the generator is pinned by version **and** by a checksum verified before install, declared once across the workflows that install it. The deploy is a dispatch carrying an environment choice, with concurrency keyed on the **environment** and `cancel-in-progress: false`, and production gated to the default branch while any ref may reach a non-production environment. The reusable callee re-asserts the environment name in a job of its own. The upload targets a per-release directory and carries no delete flag at the environment root, and the pointer flip is a separate step. The terminal check asserts the environment, then the release id, then the URL contract, waiting for convergence to a bounded timeout rather than sampling once, and reporting an unreachable host distinctly from an HTTP status (D4.6). Retention is bounded by a declared count and one side is recorded as owning the prune: the deploy asserts it where the credential can observe the destination, and the host owns it where the credential is confined write-only (D5.6). -## Verdict +### 5B. End-to-End Trace Scenarios (No Execution, Deterministic from the YAML) -Record the workflow operational when every applicable 5A item passes, every applicable 5B scenario's predicted output equals the expected, and no 5C probe that was run contradicts either. Any applicable mismatch is a defect. The verdict names the failing guarantees with the triggering input for each, the items recorded N/A, and the 5C probes prepared but not run, so a static-only audit and a fully probed one do not read alike. Per-project-type walkthroughs mapping scenarios onto targets, including source-only, static-site, and operational shapes, are `WORKFLOW.md` section 6. +For each *applicable* scenario, evaluate every job's `if:`/`needs:` against the inputs and emit the predicted **run/skip + version + release + artifact-end-state** table, then compare to the expected. A scenario governing a construct the repo does not contain is N/A, per `WORKFLOW.md` section 1, and an absent trigger is such a construct: a dispatch-only publisher records S5, S6 and S9 N/A, since their push and schedule paths can never fire there. Minimum set: + +| # | Input | Expected output | Exercises | +| --- | --- | --- | --- | +| S1 | PR touching a build target | `changes` flags it; validation runs; that target's smoke build runs; no push, **no uploads**; validate-release **skipped (smoke), succeeds**; release **skipped**; aggregator **success**; version = prerelease; no release; no dangling artifacts | D1, D2.2, D3 | +| S2 | PR changing only docs | smoke-build **skipped**, validation runs, aggregator **success** | D1.1, D1.2, D1.5 | +| S3 | PR changing only `.github/workflows/**` | the filter marks no target -> smoke-build **skipped**, validation runs, aggregator **success** | D1.2, D1.4, D1.5 | +| S4 | PR base = default branch, carrying a build target | smoke versions as prerelease, validate-release **skipped (smoke), succeeds** so the default-branch arm does **not** fire, aggregator **success**, promotion not blocked | D1.5, D2.2, D3.2 | +| S5 | bot push to `main` not touching a release path (e.g. an Actions bump) | the paths filter excludes it, so nothing publishes | D4.1 | +| S6 | code-affecting **bot** push to `main` (a human push/promotion, or any develop push, does not) | the `plan` job gates it to the App/Dependabot actor, and `main` publishes a release | D3, D4 | +| S7 | publish run (schedule, a bot push to main, or a dispatch) | builds the **one** trigger branch: `main` -> `X.Y.Z`, `prerelease=false`, registry stable, readme run; `develop` -> `X.Y.Z-g`, `prerelease=true`, registry prerelease; `release-asset-*` consumed-then-deleted; each package build-artifact (`nuget-build-*`, `pypi-build-*`) deleted after its publish; **no dangling artifacts** | D3, D4, D5, D6, D7 | +| S8 | dispatch from a ref other than `main` or `develop` | **fails fast** | D2.3 | +| S9 | re-run publish on a schedule or push trigger, version unchanged (a dispatch re-run refreshes the release instead, per D4.4) | release-create **skipped**, `release-asset-*` delete **skipped**; NuGet/PyPI pushes no-op (server dedupe); **package build-artifacts still deleted** (their download succeeded); **Docker still re-pushes** the image; no duplicate release | D4.4, D5.2 | +| S10 | branch/version classification disagree | validate-release **fails loud**, build/publish skip | D2.2 | +| S11 | scheduled upstream-version bump (wrapper) | resolver detects a change -> commits the state file -> opens a `-` PR -> merge-bot auto-merges -> the `main` pin publishes via the gate (a develop pin does not auto-publish, shipping instead via a develop dispatch or promotion) | D8.3, D3.5 | +| S12 | deploy dispatch naming an environment | the ref gate runs **first** (production from the default branch only, any ref to a non-production environment); validation runs; the callee re-asserts the environment name; a release installs under its own id; the pointer flips as a separate step; retention is bounded by whichever of the two D5.6 shapes the repo uses, so a deploy whose credential can observe the destination asserts the count converged and one confined write-only leaves it to the host; the live check asserts the environment and the release id, waiting out the reload, then the URL contract; **no tag and no release are created** | D2.1, D4.6, D5.6 | +| S13 | deploy dispatch of a production environment from a non-default ref | **fails fast**, before anything is installed or written | D2.1 | + +### 5C. Live Probe (Where Warranted) + +Every probe here that dispatches a workflow or re-runs a real publish is the maintainer's to run, and the agent prepares the command and reads the result back afterwards. A harness that refuses such a write is the harness working as intended, and the refusal is neither re-shaped into a raw API call nor talked around (GOVERNANCE.md "Repository Boundaries and Write Safety"). + +- Open a trivial-change PR touching one target and confirm S1. *Caveat: the Docker leg logs in to the registry even on smoke and reads the buildcache, so it needs `DOCKER_HUB_*` secrets and cannot run on a fork PR (same-repo only).* +- Per registry: after a real publish, query NuGet.org for the expected version + prerelease classification (and the `.snupkg` on the symbol server), and confirm a re-run added no duplicate. For PyPI read the built `dist/*` filenames out of the build job's log, `.dev0` off `develop` vs a plain version on the default branch. +- Inspect the latest real publish's logs for `PublicRelease`/`SemVer2` per leg and confirm the artifact lifecycle (uploaded, consumed, deleted, with none left behind). +- **The deploy ref gate (S13) is verified only by tripping it.** Dispatch the production environment from a non-default ref and expect the run to fail at the gate. The evidence is four things, and each of them matters: the gate job's conclusion, its error text naming the expected and the received ref, every downstream job recorded as **skipped** rather than passed, and the production environment's deployment list carrying no deployment from the dispatched ref. Capture all four, because a gate that fails open and a gate nobody tripped produce the same empty run history, so "we have never seen it fail" is not evidence about the one control standing between a mis-dispatch and the live site. **The agent prepares the command and reads all four back afterwards. It does not fire it.** The same split applies to any probe that acts on the deploy host directly, an outbound SSH exercising a forced command among them. + +### Assessment + +Record the workflow **operational** when every *applicable* 5A item passes, every *applicable* 5B scenario's predicted output equals the expected, and no 5C probe that was run contradicts either. N/A items are excluded, never counted as failures. Any *applicable* mismatch is a **defect** -> **not operational**. Procedure: + +1. **Audit** with 5A. Record pass/fail/N-A with `file:line`. +2. **Trace** the applicable S-scenarios with 5B. Diff predicted vs expected. +3. **Probe** with 5C where a live signal exists that the static trace cannot produce: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. +4. **Verdict:** operational / not operational, with the failing guarantee(s) and the triggering input for each, the list of items recorded N/A, and the 5C probes prepared but not run. + +`WORKFLOW.md` section 5 keeps the test methodology, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/test-methodology.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. + + diff --git a/WORKFLOW.md b/WORKFLOW.md index 2c5909bf7..197dff1c0 100644 --- a/WORKFLOW.md +++ b/WORKFLOW.md @@ -31,7 +31,7 @@ The guarantees are distilled from failures observed in practice. Section 4's pre ### Branch Model -Two workflow models, set per repo by the registry `workflowModel` field. `release` (default) is the feature-branch pipeline this document specifies: +Two workflow models, set per repo by the registry `workflowModel` field. `release` (default) is the feature-branch pipeline `WORKFLOW.md` specifies: ```mermaid flowchart LR @@ -49,9 +49,9 @@ flowchart LR develop -->|merge commit, enforced lint CI| main ``` -The direct commit is an **allowance, not a substitute for review**. The ruleset drops the pull-request *requirement*, which permits a direct push without withdrawing the pull request, so a change worth reviewing still takes one and both paths reach `develop` legally. Which changes those are is stated as a shape rather than a line count in [GOVERNANCE.md "Operational Repositories"][governance-operational-repositories], which owns the test and is the one place it is written, since nothing in a ruleset can apply it. What differs is when validation lands. On the direct-commit path the commit is already on the branch, so CI can only be advisory after the fact, and that is the accepted cost of the model. On the pull-request path the change has not landed, so validation is pre-merge and actionable, which is the moment it is worth the most, and the lint workflow's `pull_request` trigger therefore names `develop` alongside `main` (Section 6). That is what makes **D1.2** hold here, since its input is *any* PR and the operational model is no exception. The check is reported on a `develop` PR rather than required, because a required status check on `develop` binds the direct push too and would dissolve the allowance the model is built on. +The direct commit is an **allowance, not a substitute for review**. The ruleset drops the pull-request *requirement*, which permits a direct push without withdrawing the pull request, so a change worth reviewing still takes one and both paths reach `develop` legally. Which changes those are is stated as a shape rather than a line count in `GOVERNANCE.md` "Operational Repositories", which owns the test and is the one place it is written, since nothing in a ruleset can apply it. What differs is when validation lands. On the direct-commit path the commit is already on the branch, so CI can only be advisory after the fact, and that is the accepted cost of the model. On the pull-request path the change has not landed, so validation is pre-merge and actionable, which is the moment it is worth the most, and the lint workflow's `pull_request` trigger therefore names `develop` alongside `main` (`WORKFLOW.md` section 6). That is what makes **D1.2** hold here, since its input is *any* PR and the operational model is no exception. The check is reported on a `develop` PR rather than required, because a required status check on `develop` binds the direct push too and would dissolve the allowance the model is built on. -Their CI is lint/validation only (editorconfig/EOL plus domain linters such as Home Assistant or ESPHome config validation or a firmware build, but **no unit tests**), so the D-guarantees below that assume a build/test pipeline are **N/A** exactly as for `source-only` (Section 6). What binds: the promotion gate, where the `develop -> main` PR must pass the required `Check pull request workflow status job`, and the source-only release on manual dispatch (`releaseTrigger: dispatch-only`; tag + source zip). Branch-model rulesets are specified in [GOVERNANCE.md "Branching Model"][governance-branching-model], not here. +Their CI is lint/validation only (editorconfig/EOL plus domain linters such as Home Assistant or ESPHome config validation or a firmware build, but **no unit tests**), so the D-guarantees in `WORKFLOW.md` section 4 that assume a build/test pipeline are **N/A** exactly as for `source-only` (`WORKFLOW.md` section 6). What binds: the promotion gate, where the `develop -> main` PR must pass the required `Check pull request workflow status job`, and the source-only release on manual dispatch (`releaseTrigger: dispatch-only`; tag + source zip). Branch-model rulesets are specified in `GOVERNANCE.md` "Branching Model" rather than in `WORKFLOW.md`. ### Two Layers: Orchestration vs Build @@ -128,7 +128,9 @@ Pick each output's path by **where the artifact goes**: - **Package-registry push** (NuGet, PyPI): the leaf builds and uploads a build artifact (`nuget-build-` / `pypi-build-`), and a separate `publish-` job in the **publishing repository's own** publisher consumes it and pushes. Both registries publish through OIDC Trusted Publishing, never a stored API key, and two things put that push outside the leaf. Trusted publishing validates the OIDC token's `job_workflow_ref` claim, which names the workflow the job actually ran from, so a push made from a reusable workflow a *different* repository hosts is rejected at the token exchange, NuGet.org answering `HTTP 401` with `does not start with //.github/workflows/`. That alone rules out a leaf another repository hosts. A leaf this repository hosts clears the claim, and the split still applies to it, because a called job declaring no `permissions:` runs under the calling job's whole grant, so a push anywhere inside the release task would put `id-token: write` on every job in it rather than at the one entry point D7.2 requires. The registered trusted-publishing policy therefore names the publisher, `publish-release.yml`. PyPI additionally gates its publish job behind an environment. NuGet.org binds its policy to the workflow file rather than to an environment and needs none. NuGet's leaf also uploads a `release-asset-*` carrying the package, and PyPI contributes none. - **Image-registry push** (Docker): the leaf pushes the default branch multi-arch (amd64+arm64) and any other branch `amd64`-only (arm64 emulation is reserved for the released image), and contributes no `release-asset-*`. - **Filesystem on a host the project owns** (a static site, a config tree): the leaf builds the tree, ships it to the host, and contributes no `release-asset-*`. The transport is the repo's own. What the contract fixes is that the deploy is a **separate `workflow_dispatch`** from the release, so a redeploy of an unchanged commit mints no tag and a host rebuild, a rollback, or proving a branch on a non-production environment costs nothing; that its credentials come from a **per-environment GitHub Environment** rather than the repository secret store; and that the deploy ends by asserting **what the host serves** rather than the transport's exit status (D4.6). Retention at the destination is bounded by a declared count with one side recorded as owning the prune, which is the deploy where its credential can observe the destination and the host where that credential is deliberately write-only (D5.6). -- **No file target via the release task** (Docker-only, PyPI-only, source-only): the release is tag + source zip + README + LICENSE. The caller **MUST pass `expect_release_assets: false`** to the release task. A publisher with file targets retains the default `true`. This setting is caller-specific. The default `true` fails on `fail_on_unmatched_files` when no assets exist. A **source-only** repo also passes every `enable_*` input as false because it has no build leaf (see Section 6). +- **No file target via the release task** (Docker-only, PyPI-only, source-only): the release is tag + source zip + README + LICENSE. The caller **MUST pass `expect_release_assets: false`** to the release task. A publisher with file targets retains the default `true`. This setting is caller-specific. The default `true` fails on `fail_on_unmatched_files` when no assets exist. A **source-only** repo also passes every `enable_*` input as false because it has no build leaf (see `WORKFLOW.md` section 6). + +`WORKFLOW.md` section 3 keeps the architecture, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/architecture.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. ## 4. Behavioral Contract: Expected Outcomes @@ -211,7 +213,7 @@ The required behaviors, organized by domain. Each is a **MUST**, stated as the o ## 5. Test Methodology -An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (section 1): a check or scenario for an absent construct is recorded N/A, not failed. +An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a check or scenario for an absent construct is recorded N/A, not failed. ### 5A. Static Audit (No Execution) @@ -219,7 +221,7 @@ Read the workflow files, `version.json`, and whatever else a check below names a **Core (every repo):** -- **D1:** a `changes` paths-filter job exists wherever the repo has a smoke build, with one entry per target naming the paths that target is built from, so a change touching no target marks nothing (a filter written as a negation instead marks a docs-only change and fails D1.1); the PR entry workflow's smoke call sets every publish flag its release task declares to false (`github`/`dockerhub`, and a package-push flag there is itself a finding, per section 1); a pushing leaf receives `smoke: true` and a derived `push` (false on smoke), and a build-only leaf receives `smoke: true` with no `push` to derive; every `upload-artifact` the smoke call reaches, in a build task and in any job collecting other jobs' artifacts alike, is gated on smoke being false, written `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings; the aggregator runs under `if: always()`, `needs:` the validation job, and the `changes` and `smoke-build` jobs too wherever the repo has a smoke build, blocks on `failure`/`cancelled`, and passes on a **skipped smoke build**, so a no-build repo's aggregator, having only the validation job to read, requires that job to have succeeded; the aggregator's own job `name:` is the string the branch ruleset's required-check `context:` carries; a validation job runs unconditionally. +- **D1:** a `changes` paths-filter job exists wherever the repo has a smoke build, with one entry per target naming the paths that target is built from, so a change touching no target marks nothing (a filter written as a negation instead marks a docs-only change and fails D1.1); the PR entry workflow's smoke call sets every publish flag its release task declares to false (`github`/`dockerhub`, and a package-push flag there is itself a finding, per `WORKFLOW.md` section 1); a pushing leaf receives `smoke: true` and a derived `push` (false on smoke), and a build-only leaf receives `smoke: true` with no `push` to derive; every `upload-artifact` the smoke call reaches, in a build task and in any job collecting other jobs' artifacts alike, is gated on smoke being false, written `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings; the aggregator runs under `if: always()`, `needs:` the validation job, and the `changes` and `smoke-build` jobs too wherever the repo has a smoke build, blocks on `failure`/`cancelled`, and passes on a **skipped smoke build**, so a no-build repo's aggregator, having only the validation job to read, requires that job to have succeeded; the aggregator's own job `name:` is the string the branch ruleset's required-check `context:` carries; a validation job runs unconditionally. - **D1.6:** the validator the repo's validation job reaches collects coverage and uploads it, in every C# and Python repo that has tests. Its C# leg runs `dotnet test --coverage --coverage-output-format cobertura --results-directory ./coverage`, with `--coverage-output` unset, wherever the repo's test project is MTP-based. An MTP-based repo also ships a root `global.json` declaring the Microsoft.Testing.Platform runner, references `Microsoft.Testing.Extensions.CodeCoverage` at 18.9.0 or later in place of `coverlet.collector`, and carries no `xunit.runner.visualstudio`, while a repo still on the VSTest collector keeps its existing validator pin, a migration owed rather than drift. Its Python leg runs `pytest --cov-report=xml`, and the repo declares `pytest-cov` among its test dependencies, a dev dependency group in a uv project and a `requirements*.txt` entry on pip, selects the coverage source in its own `pyproject.toml` rather than leaving `--cov` unset, and leaves the report at the repo root as `coverage.xml`. Either way the report reaches a `codecov/codecov-action` step made best-effort by `continue-on-error` and/or `fail_ci_if_error: false`, and `CODECOV_TOKEN` is present in **both** the repo's Actions and its Dependabot secret names, the second because a Dependabot-triggered run reads that store and the upload otherwise skips silently on every bot pull request. `codecov.yml` sets the project and patch statuses `informational: true`, or names the threshold the repo enforces instead, and lists any intentionally-untested, non-shipped project under `ignore`. `.gitignore` excludes the coverage output. Record the whole item N/A for a repo with no tests, and for a type the audited repo carries at the `lint-only` profile. - **D2:** an entry validation job/step exists per complex-input workflow; the release gate checks both directions, strips `+buildmetadata`, and skips on smoke; the publisher rejects a dispatch from a ref other than `main` or `develop`. - **D3:** each run builds one branch, so NBGV classifies `github.ref` directly (no `IGNORE_GITHUB_REF`), and the default-branch literal in the gate (`== 'main'`), the `prerelease` expression (`!= 'main'`), and `version.json`'s `publicReleaseRefSpec` all name the repo's actual default branch. `version.json` sets the major.minor floor, and NBGV and `version.json` are both retained even by a repo with no compiler, since they own the tag (D3.3). NuGet.org derives the prerelease flag from the SemVer2 `-g` suffix rather than the workflow setting one, and the PyPI version is built from `AssemblyFileVersion` with `.dev0` appended on `develop` only (D3.4). @@ -239,7 +241,7 @@ Read the workflow files, `version.json`, and whatever else a check below names a ### 5B. End-to-End Trace Scenarios (No Execution, Deterministic from the YAML) -For each *applicable* scenario, evaluate every job's `if:`/`needs:` against the inputs and emit the predicted **run/skip + version + release + artifact-end-state** table, then compare to the expected. A scenario governing a construct the repo does not contain is N/A, per section 1, and an absent trigger is such a construct: a dispatch-only publisher records S5, S6 and S9 N/A, since their push and schedule paths can never fire there. Minimum set: +For each *applicable* scenario, evaluate every job's `if:`/`needs:` against the inputs and emit the predicted **run/skip + version + release + artifact-end-state** table, then compare to the expected. A scenario governing a construct the repo does not contain is N/A, per `WORKFLOW.md` section 1, and an absent trigger is such a construct: a dispatch-only publisher records S5, S6 and S9 N/A, since their push and schedule paths can never fire there. Minimum set: | # | Input | Expected output | Exercises | | --- | --- | --- | --- | @@ -275,6 +277,8 @@ Record the workflow **operational** when every *applicable* 5A item passes, ever 3. **Probe** with 5C where a live signal exists that the static trace cannot produce: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. 4. **Verdict:** operational / not operational, with the failing guarantee(s) and the triggering input for each, the list of items recorded N/A, and the 5C probes prepared but not run. +`WORKFLOW.md` section 5 keeps the test methodology, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/test-methodology.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. + ## 6. Per-Project-Type Test Walkthroughs Each type maps the *applicable* S-scenarios onto its targets. The differences are which leaf tasks exist and what each produces, which 5A addenda apply, and which scenarios are N/A. Walking these is the self-check that the contract holds for each shape. @@ -292,4 +296,3 @@ Each type maps the *applicable* S-scenarios onto its targets. The differences ar [codestyle]: ./CODESTYLE.md [governance-branching-model]: ./GOVERNANCE.md#branching-model -[governance-operational-repositories]: ./GOVERNANCE.md#operational-repositories diff --git a/docs/fleet-map.md b/docs/fleet-map.md index d04d2f046..e38e64c5a 100644 --- a/docs/fleet-map.md +++ b/docs/fleet-map.md @@ -227,7 +227,7 @@ flowchart LR ### G9: WORKFLOW.md and AUDIT.md Have No Skill (Closed) - **Gap** - The largest law doc ([`WORKFLOW.md`][workflow], the D1-D9 contract) and the measurement procedure ([`AUDIT.md`][audit]) had no skill surface, while every other procedure and language did. Thirteen [`GOVERNANCE.md`][governance] sections were likewise doc-only. -- **Resolution** - `audit-a-repo` packages `AUDIT.md` in the kept-authority shape, the doc keeping the full rules and the skill routing into it. `workflow-ci-contract` packages `WORKFLOW.md` section 4 as a generated include and the rest of that document in the same kept-authority shape. The [`AGENTS.md`][agents] rule map carries a disposition per section: `Workflow YAML Conventions` and the three conduct sections are annotated with their surfacing skill, and a paragraph after the table states why each remaining unannotated section is doc-only by decision, so absence reads as a choice rather than an oversight. Both closing tests hold: the skills ship, and the map carries the dispositions. +- **Resolution** - `audit-a-repo` packages `AUDIT.md` in the kept-authority shape, the doc keeping the full rules and the skill routing into it. `workflow-ci-contract` packages `WORKFLOW.md` sections 3, 4, and 5 as generated includes and the rest of that document in the same kept-authority shape. The [`AGENTS.md`][agents] rule map carries a disposition per section: `Workflow YAML Conventions` and the three conduct sections are annotated with their surfacing skill, and a paragraph after the table states why each remaining unannotated section is doc-only by decision, so absence reads as a choice rather than an oversight. Both closing tests hold: the skills ship, and the map carries the dispositions. - **Provenance** - All four phase-2 skills shipped in one pull request at the maintainer's direction, superseding the one-pull-request-per-skill note this doc carried, with `skill-lifecycle` authored first inside it so the others follow its procedure. ### G10: The Skill Lifecycle Has No Skill (Closed) @@ -264,10 +264,10 @@ Four skills close G9, G10, and G12, shipped through the [`.agents/skills/`][skil ### workflow-ci-contract -- **Scope** - The [`WORKFLOW.md`][workflow] behavioral contract: the D-guarantees, the seam contract, artifact lifecycle, NBGV versioning, validate-at-entry, and the per-type walkthroughs as references. +- **Scope** - The [`WORKFLOW.md`][workflow] behavioral contract: the D-guarantees, the seam contract, artifact lifecycle, NBGV versioning, and validate-at-entry, with the architecture, the guarantee catalog, and the test methodology carried as references. - **Trigger** - Writing or editing workflow YAML, adding or dropping a release target, or reasoning about why a publish did or did not fire. - **Packages** - The YAML half of the pipeline. `operational-vs-release-workflow` keeps the git half (branching, promotion, publish policy), and the two descriptions state the split. -- **Overlap** - The source doc is large, so the skill is a summary with `references/` splits, the shape `comment-and-doc-style` already uses. Section 4 is carried whole as a generated include and section 5 is indexed. +- **Overlap** - The source doc is large, so the skill is a summary with `references/` splits, the shape `comment-and-doc-style` already uses. Sections 3, 4, and 5 are each carried whole as a generated include. ### skill-lifecycle diff --git a/reports/canonical-review.json b/reports/canonical-review.json index 2c2ea852f..27cd6a24b 100644 --- a/reports/canonical-review.json +++ b/reports/canonical-review.json @@ -483,11 +483,11 @@ }, { "unit": ".agents/skills/skill-lifecycle/SKILL.md > The Doc-Packaging Pattern", - "digest": "sha256:b22e71640b0d6582e7e7d624d03436f443e688cf192bbf4a55de6f3411fcb881", + "digest": "sha256:357a856eed4481c8c1a75b87928b00b69d5b7cdd4866b82fd84134b179138dd0", "reviewer": "agent-skill", - "findings": 0, - "hubCommit": "ee4c1ce3dc1a67520b00f2038136c252172f52f5", - "stamp": "2026-09-05T22:26:20Z" + "findings": 3, + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T17:51:06Z" }, { "unit": ".agents/skills/skill-lifecycle/SKILL.md > The Pipeline", @@ -515,11 +515,11 @@ }, { "unit": ".agents/skills/workflow-ci-contract/SKILL.md > How the Contract Is Read", - "digest": "sha256:57ad04138726b32fd827bc6717119e09839f575ae4283dcbbcf7736b40637857", + "digest": "sha256:b208f86efb05ab8e3ff139f25e4610487358fa03df0222a224c540cd14882122", "reviewer": "agent-skill", - "findings": 1, - "hubCommit": "ee4c1ce3dc1a67520b00f2038136c252172f52f5", - "stamp": "2026-09-05T22:25:58Z" + "findings": 0, + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T18:19:28Z" }, { "unit": ".agents/skills/workflow-ci-contract/SKILL.md > Style Rules", @@ -529,6 +529,14 @@ "hubCommit": "66d79460335379301db83fe6870303f8f701d92b", "stamp": "2026-09-06T00:53:56Z" }, + { + "unit": ".agents/skills/workflow-ci-contract/SKILL.md > The Contract Text", + "digest": "sha256:4a97612e6fc099699e97f76267186932eba13968557175e0b8534e493b247b1d", + "reviewer": "agent-skill", + "findings": 2, + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T17:51:06Z" + }, { "unit": ".agents/skills/workflow-ci-contract/SKILL.md > The Core Behavioral Spine", "digest": "sha256:c67f5b8b324d7a796098ba4bb04174b1c07b7da05627416cefe7d149a93a1719", @@ -539,19 +547,35 @@ }, { "unit": ".agents/skills/workflow-ci-contract/SKILL.md > Why This Exists", - "digest": "sha256:b7dfbb5b5d20d1bb2c1e78c3a4a0e61256dc4e5a88c4e68169968d42e534d6de", + "digest": "sha256:d89e6f3b20c6ec6ca8307fae5346303fc0e0c9eb439f3b960b75f0733a7d836c", "reviewer": "agent-skill", "findings": 0, - "hubCommit": "ee4c1ce3dc1a67520b00f2038136c252172f52f5", - "stamp": "2026-09-05T19:41:37Z" + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T17:51:06Z" + }, + { + "unit": ".agents/skills/workflow-ci-contract/references/architecture.md > (preamble)", + "digest": "sha256:6d9507914d458b12d94684f297b13de873beae3f48dc0a040a5757ca90e8b566", + "reviewer": "agent-skill", + "findings": 3, + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T17:51:06Z" + }, + { + "unit": ".agents/skills/workflow-ci-contract/references/architecture.md > The Architecture", + "digest": "sha256:32d3c4d8f85423bad4dc0066f8e93e90f27d1dcc172b0b06ec8d826031c132c7", + "reviewer": "agent-skill", + "findings": 2, + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T17:51:06Z" }, { "unit": ".agents/skills/workflow-ci-contract/references/d-guarantees.md > (preamble)", - "digest": "sha256:a8eca6119c9f149c6998f5c57ca73b083c7cc79b36bf5536f67d1f27ec6cfb14", + "digest": "sha256:62ab330411b5d4fb24146921d173bf78005af48b9abffb00dce980f06947ae87", "reviewer": "agent-skill", "findings": 0, - "hubCommit": "ee4c1ce3dc1a67520b00f2038136c252172f52f5", - "stamp": "2026-09-05T19:40:55Z" + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T18:08:47Z" }, { "unit": ".agents/skills/workflow-ci-contract/references/d-guarantees.md > The Behavioral Contract", @@ -561,6 +585,14 @@ "hubCommit": "66d79460335379301db83fe6870303f8f701d92b", "stamp": "2026-09-06T00:46:52Z" }, + { + "unit": ".agents/skills/workflow-ci-contract/references/test-methodology.md > (preamble)", + "digest": "sha256:9429013b2fc946cf54d1eb40ca5e3756211ca3e7992911ac50fb837e325e3771", + "reviewer": "agent-skill", + "findings": 2, + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T17:51:06Z" + }, { "unit": ".agents/skills/workflow-ci-contract/references/test-methodology.md > 5A: Static Audit", "digest": "sha256:d3dff5e85fa809cd530b313d0e108f260d7f5437274d5261aeafde77d13666e7", @@ -585,6 +617,14 @@ "hubCommit": "28dafdf46ee35cc9901e845a317991d909a94fac", "stamp": "2026-09-03T18:19:22Z" }, + { + "unit": ".agents/skills/workflow-ci-contract/references/test-methodology.md > The Test Methodology", + "digest": "sha256:a084b10818b512d3c6696aae3d678955cbe27290631046e173d2cdd035b59cc1", + "reviewer": "agent-skill", + "findings": 0, + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T17:51:07Z" + }, { "unit": ".agents/skills/workflow-ci-contract/references/test-methodology.md > Verdict", "digest": "sha256:fca3f384bfead20bbf4da5381ada9abf417451d72f8e88a51a246fa910faf4a2", @@ -731,11 +771,11 @@ }, { "unit": "WORKFLOW.md > 3. Architecture", - "digest": "sha256:cbdfc5161d7951fc9f968257f9090aa8086f10f5fcb869d3a632d12ccfdb2dbe", + "digest": "sha256:c0f4d188701cb09ca06f80831d40f4c0cecc8ca75b08fe44ac62aa27d2dd06e0", "reviewer": "agent-skill", - "findings": 62, - "hubCommit": "9b33edb7e0f55fe2a05630a49d5be99ba1e3eb26", - "stamp": "2026-09-02T15:16:10Z" + "findings": 10, + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T17:51:07Z" }, { "unit": "WORKFLOW.md > 4. Behavioral Contract: Expected Outcomes", @@ -747,19 +787,19 @@ }, { "unit": "WORKFLOW.md > 5. Test Methodology", - "digest": "sha256:a7dff7c97f08703ef7d9870e3bef1e368d37af4743f51dd1bafbef7beaf63f45", + "digest": "sha256:0bdfdb5d7fc20469237f06286a59d36d245192c15310ae55b245196f13206eec", "reviewer": "agent-skill", - "findings": 2, - "hubCommit": "66d79460335379301db83fe6870303f8f701d92b", - "stamp": "2026-09-06T02:19:37Z" + "findings": 6, + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T17:51:07Z" }, { "unit": "WORKFLOW.md > 6. Per-Project-Type Test Walkthroughs", - "digest": "sha256:b2967d951f386797e8c5e5b4b3e4e1699c81e45e72650ff1e3f4d0e4f28e47dd", + "digest": "sha256:ade51d875b83dfe761ec121d307a06c21e9ba730eaa2ff27e4e428833ee8cfdf", "reviewer": "agent-skill", - "findings": 3, - "hubCommit": "66d79460335379301db83fe6870303f8f701d92b", - "stamp": "2026-09-06T00:46:51Z" + "findings": 5, + "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", + "stamp": "2026-09-06T17:51:07Z" } ] }