diff --git a/.agents/skills/audit-a-repo/SKILL.md b/.agents/skills/audit-a-repo/SKILL.md index 5cd0c133..070703f9 100644 --- a/.agents/skills/audit-a-repo/SKILL.md +++ b/.agents/skills/audit-a-repo/SKILL.md @@ -18,10 +18,10 @@ The audit is the fleet's measurement procedure, and the two failure shapes it gu ## Measuring -- **Resolve the repo's types from `registry/repos.json`** and classify a `classificationPending` entry from the tree (`AUDIT.md` section 2). The applicability gate is `WORKFLOW.md` section 1: a check governing an absent construct is N/A, excluded from the verdict, and never a defect (`AUDIT.md` section 3). +- **Resolve the repo's types from `registry/repos.json`** and classify a `classificationPending` entry from the tree (`AUDIT.md` section 2). The applicability gate is `WORKFLOW.md` section 1, extended to `AUDIT.md`'s own checks: an item or check governing an absent construct is N/A, excluded from the verdict, and never a defect (`AUDIT.md` section 3). - **Know what the runner does and does not prove.** `spec/audit.py` mechanizes the deterministic subset only: settings, rulesets, secret names, file and section presence, verbatim hashing, interface wiring, Dependabot coverage, branch facts. It evaluates no check under a type in `spec/project-types.json`, so every per-type check is judged by hand, and a clean run is no evidence for them (`AUDIT.md` section 4). Silence from a tool that was never looking reads exactly like a pass. - **Judge letter and intent per check** and keep the vocabulary: letter miss with intent satisfied is a drift finding, both missing is a defect, and operational is binary over the applicable set (`AUDIT.md` sections 4 and 7). Do not invent a parallel scheme. -- **Assert the Actions implement `WORKFLOW.md`** by outcome, not by matching catalog snippets byte for byte: the 5A static audit with a `file:line` citation per applicable guarantee, then the 5B trace scenarios (`AUDIT.md` section 5). The `workflow-ci-contract` skill summarizes that contract. +- **Assert the Actions implement `WORKFLOW.md`** by outcome, not by matching catalog snippets byte for byte: the 5A static audit, each applicable guarantee cited in the form 5A sets out, then the 5B trace scenarios (`AUDIT.md` section 5). Read a workflow the repo only calls at the SHA it pins, for both. The `workflow-ci-contract` skill summarizes that contract. - **Check live settings, rulesets, and secrets from a hub checkout at `main`** with `AUDIT.md` section 6. Run `repo-config/configure.sh check` with the target repository and model for settings and rulesets, and `spec/audit.py [RepoName]` for secrets, rather than constructing a local comparison. The hub payloads are the only repository-configuration source. ## Reporting diff --git a/.agents/skills/workflow-ci-contract/SKILL.md b/.agents/skills/workflow-ci-contract/SKILL.md index 8f751413..a6fe9ab2 100644 --- a/.agents/skills/workflow-ci-contract/SKILL.md +++ b/.agents/skills/workflow-ci-contract/SKILL.md @@ -13,10 +13,10 @@ description: >- ## How the Contract Is Read - **Outcomes, not bytes.** A workflow is judged against `WORKFLOW.md` section 4's expected inputs and outputs, never against a snippet byte for byte, per `GOVERNANCE.md` "Foundational Principles". -- **Applicability.** A guarantee, or a 5A check or 5B scenario from `WORKFLOW.md` section 5, governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. +- **Applicability.** A guarantee, or a 5B scenario from `WORKFLOW.md` section 5, governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. - **Operational is binary.** Every applicable guarantee holds, or the workflow is not operational. A single applicable input-output mismatch is a defect regardless of how clean the YAML looks. - **Reached, not carried.** A standard workflow whose job graph is identical across repos of a type is reached as a hub-hosted `workflow_call` task, per `GOVERNANCE.md` "Hub-Hosted Tooling". The repo's own surface is the caller stub, pinned to a hub release commit, and a composite-action hook at `.github/actions/` for what is its own. A hub task reaches its own actions and sibling tasks through `$/`, which resolves at that pinned commit. The merge-bot is the first, and `docs/reusable-workflows.md` in the hub carries the model, the hook contract, and the stage each workflow migrates in. Until a workflow's stage ships, its copy is graded against the same contract. -- **Two layers.** The pipeline splits into an orchestrator layer and a build-leaf layer, defined in `WORKFLOW.md` section 3's `Two Layers: Orchestration vs Build` and carried in `references/architecture.md`, while `WORKFLOW.md` section 1's `Two layers when auditing` maps which layer declares which input. When a check names an input, assert it in the layer that declares it. +- **Two layers.** The pipeline splits into an orchestrator layer and a build-leaf layer, defined in `WORKFLOW.md` section 3's `Two Layers: Orchestration vs Build` and carried in `references/architecture.md`, while `WORKFLOW.md` section 1's `Two layers when auditing` maps which layer declares which input. Assert an input a guarantee names in the layer that declares it. ## Style Rules diff --git a/.agents/skills/workflow-ci-contract/references/test-methodology.md b/.agents/skills/workflow-ci-contract/references/test-methodology.md index 744bff09..1dd10c14 100644 --- a/.agents/skills/workflow-ci-contract/references/test-methodology.md +++ b/.agents/skills/workflow-ci-contract/references/test-methodology.md @@ -1,36 +1,20 @@ # Testing a Repo's Workflows -The section below is `WORKFLOW.md` section 5, whole. Which of its checks and scenarios bind a given repository is `WORKFLOW.md` section 1's applicability rule, and the D-guarantees each one answers to are `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. +The section below is `WORKFLOW.md` section 5, whole. Its items and scenarios answer to the D-guarantees in `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. ## The Test Methodology -An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a check or scenario for an absent construct is recorded N/A, not failed. +An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a guarantee or scenario for an absent construct is recorded N/A, not failed. ### 5A. Static Audit (No Execution) -Read the workflow files, `version.json`, and whatever else a check below names as its evidence: a project or dependency file, `global.json`, `codecov.yml`, `.gitignore`, the branch ruleset, and the repo's Actions and Dependabot secret names. Assert the structural fact behind each *applicable* D-guarantee, each pass/fail/N-A with a `file:line` citation, and cite a repository setting by its own name where that setting rather than a file is the evidence. Remember the two layers: assert each input in the layer that declares it. +Assert the structural fact each *applicable* D-guarantee implies, and record **pass**, **fail**, or **N/A** per item. This section says how an audit is run and recorded rather than what must hold: a guarantee names its own constructs, and the requirement is `WORKFLOW.md` section 4's item together with whatever that item defers to. -**Core (every repo):** +Most of the evidence is in the workflow files and the composite actions they reach. Where a guarantee's evidence lies outside them, it is in practice the repo's branch ruleset, its Actions and Dependabot secret names, a workflow the repo only calls, a project or dependency file, or a committed file such as `version.json`, `.github/dependabot.yml`, `global.json`, `codecov.yml`, `.gitignore`, or `.editorconfig`. -- **D1:** a `changes` paths-filter job exists wherever the repo has a smoke build, with one entry per target naming the paths that target is built from, so a change touching no target marks nothing (a filter written as a negation instead marks a docs-only change and fails D1.1); the PR entry workflow's smoke call sets every publish flag its release task declares to false (`github`/`dockerhub`, and a package-push flag there is itself a finding, per `WORKFLOW.md` section 1); a pushing leaf receives `smoke: true` and a derived `push` (false on smoke), and a build-only leaf receives `smoke: true` with no `push` to derive; every `upload-artifact` the smoke call reaches, in a build task and in any job collecting other jobs' artifacts alike, is gated on smoke being false, written `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings; the aggregator runs under `if: always()`, `needs:` the validation job, and the `changes` and `smoke-build` jobs too wherever the repo has a smoke build, blocks on `failure`/`cancelled`, and passes on a **skipped smoke build**, so a no-build repo's aggregator, having only the validation job to read, requires that job to have succeeded; the aggregator's own job `name:` is the string the branch ruleset's required-check `context:` carries; a validation job runs unconditionally. -- **D1.6:** the validator the repo's validation job reaches collects coverage and uploads it, in every C# and Python repo that has tests. Its C# leg runs `dotnet test --coverage --coverage-output-format cobertura --results-directory ./coverage`, with `--coverage-output` unset, wherever the repo's test project is MTP-based. An MTP-based repo also ships a root `global.json` declaring the Microsoft.Testing.Platform runner, references `Microsoft.Testing.Extensions.CodeCoverage` at 18.9.0 or later in place of `coverlet.collector`, and carries no `xunit.runner.visualstudio`, while a repo still on the VSTest collector keeps its existing validator pin, a migration owed rather than drift. Its Python leg runs `pytest --cov-report=xml`, and the repo declares `pytest-cov` among its test dependencies, a dev dependency group in a uv project and a `requirements*.txt` entry on pip, selects the coverage source in its own `pyproject.toml` rather than leaving `--cov` unset, and leaves the report at the repo root as `coverage.xml`. Either way the report reaches a `codecov/codecov-action` step made best-effort by `continue-on-error` and/or `fail_ci_if_error: false`, and `CODECOV_TOKEN` is present in **both** the repo's Actions and its Dependabot secret names, the second because a Dependabot-triggered run reads that store and the upload otherwise skips silently on every bot pull request. `codecov.yml` sets the project and patch statuses `informational: true`, or names the threshold the repo enforces instead, and lists any intentionally-untested, non-shipped project under `ignore`. `.gitignore` excludes the coverage output. Record the whole item N/A for a repo with no tests, and for a type the audited repo carries at the `lint-only` profile. -- **D2:** an entry validation job/step exists per complex-input workflow; the release gate checks both directions, strips `+buildmetadata`, and skips on smoke; the publisher rejects a dispatch from a ref other than `main` or `develop`. -- **D3:** each run builds one branch, so NBGV classifies `github.ref` directly (no `IGNORE_GITHUB_REF`), and the default-branch literal in the gate (`== 'main'`), the `prerelease` expression (`!= 'main'`), and `version.json`'s `publicReleaseRefSpec` all name the repo's actual default branch. `version.json` sets the major.minor floor, and NBGV and `version.json` are both retained even by a repo with no compiler, since they own the tag (D3.3). NuGet.org derives the prerelease flag from the SemVer2 `-g` suffix rather than the workflow setting one, and the PyPI version is built from `AssemblyFileVersion` with `.dev0` appended on `develop` only (D3.4). -- **D4:** `target_commitish` is the NBGV commit id; `prerelease` equals `branch != default`; the release-create step is gated `exists == 'false' || github.event_name == 'workflow_dispatch'` (the step output is the string `'false'`, not a boolean); the asset-delete step carries that same condition, narrowed by `inputs.expect_release_assets`. Where `workflow_dispatch` is the publisher's only trigger (`releaseTrigger: dispatch-only`) every run is a dispatch, so the exists-check's skip leg can never fire: record that leg N/A rather than failed, and expect the release-create step to still carry the `exists == 'false' || github.event_name == 'workflow_dispatch'` condition, since D6.4 keeps the `github-release` job body verbatim. A caller with no file target passes `expect_release_assets: false`, which covers a Docker-only, a PyPI-only, and a source-only repo alike, while a NuGet-only caller keeps the default `true` because its leaf uploads a `release-asset-*` carrying the package, and a source-only caller also sets every `enable_*` input false. `github-release` and the terminal registry pusher (Docker) each carry `!failure() && !cancelled()` rather than the implicit `success()`, so a failed build skips both, while a target that merely skipped, being disabled or unchanged, still lets the release and the Docker push proceed, and a package target's separate publish job `needs:` the release-task call for the same reason (D4.5). A first `plan` job decides once whether the run publishes, admitting a code-affecting bot push to `main`, a dispatch of `main` or `develop`, and a `main`-only schedule, and every publishing job gates on that decision (D4.1). -- **D5:** each cross-job transfer artifact has a delete step at its consumer, gated so it runs exactly when the consumption happened rather than when the whole job succeeded (D5.2), `continue-on-error: true`, tolerating a failed listing, and looping all ids; **every** upload sets `retention-days: 1`; **no** cleanup step enumerates and deletes the run's whole artifact set, in whatever jq or API shape it is written. -- **D6:** the release download uses `pattern:`/`merge-multiple:` (no `artifact-ids:`). Branch-derived config reads `inputs.branch` (a `github.ref_name` in such config is a finding). Artifact names are branch-suffixed. The target set is consistent across every surface D6.4 names: the `enable_` input, the `build-` job, that job's entries in the `github-release` and `build-docker` `needs:` lists, the `changes` paths-filter entry **and** its output, the `smoke-build` enable-forward, and any separate `publish-` job the package-registry seam requires. The `inputs.branch` rule above binds a called leaf, while a `publish-` job is in the publisher and reads `github.ref_name` correctly. -- **D7:** the publisher concurrency group is global and ref-independent with `cancel-in-progress: false`. A reusable job declares `permissions:` only where every caller grants that scope at startup, per D7.2. A boolean used by both `workflow_call` and `workflow_dispatch` is declared in both trigger blocks, and a boolean read through `github.event.inputs.` is compared against `'true'` alone, a comparison against the boolean `true` never firing on a string, while an `inputs.` read carries the declared boolean and is used directly, a both-forms comparison there being redundant rather than a finding, so a repo whose booleans all arrive by `workflow_call` records the comparison half N/A. A job or step output is a string too and takes the same `== 'true'` (D7.3). Every cross-job condition that admits a skipped dependency pairs its allowlist with a status-check function such as `always()` or `!failure() && !cancelled()`, since the implicit `success()` is false the moment a `needs:` job skipped (D7.4). -- **D8/D9:** the merge-bot enables auto-merge on `opened`/`reopened` for every Dependabot tier including semver-major, dispatches `--squash`/`--merge` by the PR's base ref, disables on a maintainer-pushed `synchronize`, and keys concurrency on the PR number rather than `github.ref`. Codegen runs as a matrix over both branches, and Dependabot targets both branches with security PRs to the default branch (D8.2). The upstream tracker's `bump-branch-prefix` and `branches` match a merge-bot rule, unless it sets `auto-merge: false`, which prefixes the head so no rule matches (wrapper repos). A gate comparing `github.actor` against hard-coded bot identities emits a `::warning::` on the non-matching branch, and the annotation is optional only where the failure announces itself anyway (D8.4). Actions are SHA-pinned. Names, shells, and conditionals follow `GOVERNANCE.md` "Workflow YAML Conventions", and line endings follow `.editorconfig` (D9.5), per the policy `GOVERNANCE.md` "Documentation Style Conventions" routes to under "Line Endings". - -**Per-type addenda (apply only the ones present):** - -- **.NET publish:** the smoke runtime set is a strict non-empty subset of the full runtime set. The selected set runs sequentially inside one composite-action job. A non-smoke run uploads one `release-asset--dotnet-publish` artifact, while a smoke run skips the archive and upload steps. -- **NuGet:** `publish-nuget` is a job in the repo's own publisher, never inside the release task and never in a reusable workflow a different repository hosts. `id-token: write` appears on that job only, absent from the build and PR paths, beside `actions: write` for the artifact cleanup. The push uses `--skip-duplicate` and is gated by that job's `needs:` on the release-task call, never on an existence check, so a PR never reaches it. The job consume-then-deletes `nuget-build-` under the download step's own success, per D5.1 and D5.2. `*.nupkg` push also carries the paired `.snupkg` to the symbol server where symbols are enabled. The `release-asset` `.7z` carries the package(s). -- **PyPI:** `publish-pypi` declares `environment: { name: pypi }`. `id-token: write` appears only on that job (absent from the build/PR path). `skip-existing: true` is set on the publish action. The build artifact is deleted under the download step's own success, per D5.2. The `pypi` environment has a deployment-branch rule. -- **Docker:** the leaf reads the external state file for the tag instead of `SemVer2` (wrapper repos only, since a plain Docker repo correctly tags off `SemVer2` and records this N/A). The readme job is gated main-only, both by the caller's branch input and inside the hub-hosted `publish-docker-readme-task.yml` itself. The docker-readme task validates `repositories` XOR `manifest`+`manifest-jq`. The buildcache follows D9.4. -- **Static site deployed to a host:** the generator is pinned by version **and** by a checksum verified before install, declared once across the workflows that install it. The deploy is a dispatch carrying an environment choice, with concurrency keyed on the **environment** and `cancel-in-progress: false`, and production gated to the default branch while any ref may reach a non-production environment. The reusable callee re-asserts the environment name in a job of its own. The upload targets a per-release directory and carries no delete flag at the environment root, and the pointer flip is a separate step. The terminal check asserts the environment, then the release id, then the URL contract, waiting for convergence to a bounded timeout rather than sampling once, and reporting an unreachable host distinctly from an HTTP status (D4.6). Retention is bounded by a declared count and one side is recorded as owning the prune: the deploy asserts it where the credential can observe the destination, and the host owns it where the credential is confined write-only (D5.6). +Cite what each verdict rests on. That is `file:line` for a file in the audited repo, its own name where a setting, a ruleset, or a secret name rather than a file is the evidence, and `/@` plus the `file:line` in that repo where the guarantee binds a workflow or composite action the audited repo only reaches, read at the SHA the caller pins. An **N/A** verdict names the absent construct instead, there being no line to cite. ### 5B. End-to-End Trace Scenarios (No Execution, Deterministic from the YAML) @@ -38,23 +22,23 @@ For each *applicable* scenario, evaluate every job's `if:`/`needs:` against the | # | Input | Expected output | Exercises | | --- | --- | --- | --- | -| S1 | PR touching a build target | `changes` flags it; validation runs; that target's smoke build runs; no push, **no uploads**; validate-release **skipped (smoke), succeeds**; release **skipped**; aggregator **success**; version = prerelease; no release; no dangling artifacts | D1, D2.2, D3 | +| S1 | PR touching a build target | `changes` flags it; validation runs; that target's smoke build runs; no push, **no uploads**; validate-release **succeeds**, its check exiting early on smoke per D2.2; release **skipped**; aggregator **success**; version = prerelease; no release; no dangling artifacts | D1, D2.2, D3 | | S2 | PR changing only docs | smoke-build **skipped**, validation runs, aggregator **success** | D1.1, D1.2, D1.5 | | S3 | PR changing only `.github/workflows/**` | the filter marks no target -> smoke-build **skipped**, validation runs, aggregator **success** | D1.2, D1.4, D1.5 | -| S4 | PR base = default branch, carrying a build target | smoke versions as prerelease, validate-release **skipped (smoke), succeeds** so the default-branch arm does **not** fire, aggregator **success**, promotion not blocked | D1.5, D2.2, D3.2 | +| S4 | PR base = default branch, carrying a build target | smoke versions as prerelease, validate-release **succeeds** with its check exited early per D2.2, so the default-branch arm does **not** fire, aggregator **success**, promotion not blocked | D1.5, D2.2, D3.2 | | S5 | bot push to `main` not touching a release path (e.g. an Actions bump) | the paths filter excludes it, so nothing publishes | D4.1 | | S6 | code-affecting **bot** push to `main` (a human push/promotion, or any develop push, does not) | the `plan` job gates it to the App/Dependabot actor, and `main` publishes a release | D3, D4 | | S7 | publish run (schedule, a bot push to main, or a dispatch) | builds the **one** trigger branch: `main` -> `X.Y.Z`, `prerelease=false`, registry stable, readme run; `develop` -> `X.Y.Z-g`, `prerelease=true`, registry prerelease; `release-asset-*` consumed-then-deleted; each package build-artifact (`nuget-build-*`, `pypi-build-*`) deleted after its publish; **no dangling artifacts** | D3, D4, D5, D6, D7 | | S8 | dispatch from a ref other than `main` or `develop` | **fails fast** | D2.3 | | S9 | re-run publish on a schedule or push trigger, version unchanged (a dispatch re-run refreshes the release instead, per D4.4) | release-create **skipped**, `release-asset-*` delete **skipped**; NuGet/PyPI pushes no-op (server dedupe); **package build-artifacts still deleted** (their download succeeded); **Docker still re-pushes** the image; no duplicate release | D4.4, D5.2 | | S10 | branch/version classification disagree | validate-release **fails loud**, build/publish skip | D2.2 | -| S11 | scheduled upstream-version bump (wrapper) | resolver detects a change -> commits the state file -> opens a `-` PR -> merge-bot auto-merges -> the `main` pin publishes via the gate (a develop pin does not auto-publish, shipping instead via a develop dispatch or promotion) | D8.3, D3.5 | +| S11 | scheduled upstream-version bump (wrapper) | resolver detects a change -> commits the state file -> opens a per-branch bump PR -> the merge-bot auto-merges it, or leaves it for the maintainer where the tracker sets `auto-merge: false` (D8.3) -> the `main` pin publishes via the gate (a develop pin does not auto-publish, shipping instead via a develop dispatch or promotion) | D8.3, D3.5 | | S12 | deploy dispatch naming an environment | the ref gate runs **first** (production from the default branch only, any ref to a non-production environment); validation runs; the callee re-asserts the environment name; a release installs under its own id; the pointer flips as a separate step; retention is bounded by whichever of the two D5.6 shapes the repo uses, so a deploy whose credential can observe the destination asserts the count converged and one confined write-only leaves it to the host; the live check asserts the environment and the release id, waiting out the reload, then the URL contract; **no tag and no release are created** | D2.1, D4.6, D5.6 | | S13 | deploy dispatch of a production environment from a non-default ref | **fails fast**, before anything is installed or written | D2.1 | ### 5C. Live Probe (Where Warranted) -Every probe here that dispatches a workflow or re-runs a real publish is the maintainer's to run, and the agent prepares the command and reads the result back afterwards. A harness that refuses such a write is the harness working as intended, and the refusal is neither re-shaped into a raw API call nor talked around (GOVERNANCE.md "Repository Boundaries and Write Safety"). +Every probe here that opens a pull request, dispatches a workflow, or re-runs a real publish is the maintainer's to run, with the agent preparing the command and reading the result back afterwards. A harness that refuses such a write is the harness working as intended, and the refusal is neither re-shaped into a raw API call nor talked around (`GOVERNANCE.md` "Repository Boundaries and Write Safety"). - Open a trivial-change PR touching one target and confirm S1. *Caveat: the Docker leg logs in to the registry even on smoke and reads the buildcache, so it needs `DOCKER_HUB_*` secrets and cannot run on a fork PR (same-repo only).* - Per registry: after a real publish, query NuGet.org for the expected version + prerelease classification (and the `.snupkg` on the symbol server), and confirm a re-run added no duplicate. For PyPI read the built `dist/*` filenames out of the build job's log, `.dev0` off `develop` vs a plain version on the default branch. @@ -65,9 +49,9 @@ Every probe here that dispatches a workflow or re-runs a real publish is the mai Record the workflow **operational** when every *applicable* 5A item passes, every *applicable* 5B scenario's predicted output equals the expected, and no 5C probe that was run contradicts either. N/A items are excluded, never counted as failures. Any *applicable* mismatch is a **defect** -> **not operational**. Procedure: -1. **Audit** with 5A. Record pass/fail/N-A with `file:line`. +1. **Audit** with 5A, recording each item's verdict and its evidence in the form 5A sets out. 2. **Trace** the applicable S-scenarios with 5B. Diff predicted vs expected. -3. **Probe** with 5C where a live signal exists that the static trace cannot produce: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. +3. **Probe** with 5C where a live signal exists that the static trace cannot produce, running the probes that only read and preparing the writing ones for the maintainer: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. 4. **Verdict:** operational / not operational, with the failing guarantee(s) and the triggering input for each, the list of items recorded N/A, and the 5C probes prepared but not run. `WORKFLOW.md` section 5 keeps the test methodology, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/test-methodology.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. diff --git a/.claude-plugin/fleet-skills/.source-digests/audit-a-repo b/.claude-plugin/fleet-skills/.source-digests/audit-a-repo index b822d4a0..f0178002 100644 --- a/.claude-plugin/fleet-skills/.source-digests/audit-a-repo +++ b/.claude-plugin/fleet-skills/.source-digests/audit-a-repo @@ -1 +1 @@ -162403e1f05c8750 +b31e858b76975e80 diff --git a/.claude-plugin/fleet-skills/.source-digests/workflow-ci-contract b/.claude-plugin/fleet-skills/.source-digests/workflow-ci-contract index 04f6a6b9..ea463f09 100644 --- a/.claude-plugin/fleet-skills/.source-digests/workflow-ci-contract +++ b/.claude-plugin/fleet-skills/.source-digests/workflow-ci-contract @@ -1 +1 @@ -68655ae1cf8f88bf +3d168164fc4e1f1a diff --git a/.claude-plugin/fleet-skills/skills/audit-a-repo/SKILL.md b/.claude-plugin/fleet-skills/skills/audit-a-repo/SKILL.md index 5cd0c133..070703f9 100644 --- a/.claude-plugin/fleet-skills/skills/audit-a-repo/SKILL.md +++ b/.claude-plugin/fleet-skills/skills/audit-a-repo/SKILL.md @@ -18,10 +18,10 @@ The audit is the fleet's measurement procedure, and the two failure shapes it gu ## Measuring -- **Resolve the repo's types from `registry/repos.json`** and classify a `classificationPending` entry from the tree (`AUDIT.md` section 2). The applicability gate is `WORKFLOW.md` section 1: a check governing an absent construct is N/A, excluded from the verdict, and never a defect (`AUDIT.md` section 3). +- **Resolve the repo's types from `registry/repos.json`** and classify a `classificationPending` entry from the tree (`AUDIT.md` section 2). The applicability gate is `WORKFLOW.md` section 1, extended to `AUDIT.md`'s own checks: an item or check governing an absent construct is N/A, excluded from the verdict, and never a defect (`AUDIT.md` section 3). - **Know what the runner does and does not prove.** `spec/audit.py` mechanizes the deterministic subset only: settings, rulesets, secret names, file and section presence, verbatim hashing, interface wiring, Dependabot coverage, branch facts. It evaluates no check under a type in `spec/project-types.json`, so every per-type check is judged by hand, and a clean run is no evidence for them (`AUDIT.md` section 4). Silence from a tool that was never looking reads exactly like a pass. - **Judge letter and intent per check** and keep the vocabulary: letter miss with intent satisfied is a drift finding, both missing is a defect, and operational is binary over the applicable set (`AUDIT.md` sections 4 and 7). Do not invent a parallel scheme. -- **Assert the Actions implement `WORKFLOW.md`** by outcome, not by matching catalog snippets byte for byte: the 5A static audit with a `file:line` citation per applicable guarantee, then the 5B trace scenarios (`AUDIT.md` section 5). The `workflow-ci-contract` skill summarizes that contract. +- **Assert the Actions implement `WORKFLOW.md`** by outcome, not by matching catalog snippets byte for byte: the 5A static audit, each applicable guarantee cited in the form 5A sets out, then the 5B trace scenarios (`AUDIT.md` section 5). Read a workflow the repo only calls at the SHA it pins, for both. The `workflow-ci-contract` skill summarizes that contract. - **Check live settings, rulesets, and secrets from a hub checkout at `main`** with `AUDIT.md` section 6. Run `repo-config/configure.sh check` with the target repository and model for settings and rulesets, and `spec/audit.py [RepoName]` for secrets, rather than constructing a local comparison. The hub payloads are the only repository-configuration source. ## Reporting diff --git a/.claude-plugin/fleet-skills/skills/workflow-ci-contract/SKILL.md b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/SKILL.md index 8f751413..a6fe9ab2 100644 --- a/.claude-plugin/fleet-skills/skills/workflow-ci-contract/SKILL.md +++ b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/SKILL.md @@ -13,10 +13,10 @@ description: >- ## How the Contract Is Read - **Outcomes, not bytes.** A workflow is judged against `WORKFLOW.md` section 4's expected inputs and outputs, never against a snippet byte for byte, per `GOVERNANCE.md` "Foundational Principles". -- **Applicability.** A guarantee, or a 5A check or 5B scenario from `WORKFLOW.md` section 5, governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. +- **Applicability.** A guarantee, or a 5B scenario from `WORKFLOW.md` section 5, governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. - **Operational is binary.** Every applicable guarantee holds, or the workflow is not operational. A single applicable input-output mismatch is a defect regardless of how clean the YAML looks. - **Reached, not carried.** A standard workflow whose job graph is identical across repos of a type is reached as a hub-hosted `workflow_call` task, per `GOVERNANCE.md` "Hub-Hosted Tooling". The repo's own surface is the caller stub, pinned to a hub release commit, and a composite-action hook at `.github/actions/` for what is its own. A hub task reaches its own actions and sibling tasks through `$/`, which resolves at that pinned commit. The merge-bot is the first, and `docs/reusable-workflows.md` in the hub carries the model, the hook contract, and the stage each workflow migrates in. Until a workflow's stage ships, its copy is graded against the same contract. -- **Two layers.** The pipeline splits into an orchestrator layer and a build-leaf layer, defined in `WORKFLOW.md` section 3's `Two Layers: Orchestration vs Build` and carried in `references/architecture.md`, while `WORKFLOW.md` section 1's `Two layers when auditing` maps which layer declares which input. When a check names an input, assert it in the layer that declares it. +- **Two layers.** The pipeline splits into an orchestrator layer and a build-leaf layer, defined in `WORKFLOW.md` section 3's `Two Layers: Orchestration vs Build` and carried in `references/architecture.md`, while `WORKFLOW.md` section 1's `Two layers when auditing` maps which layer declares which input. Assert an input a guarantee names in the layer that declares it. ## Style Rules diff --git a/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/test-methodology.md b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/test-methodology.md index 744bff09..1dd10c14 100644 --- a/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/test-methodology.md +++ b/.claude-plugin/fleet-skills/skills/workflow-ci-contract/references/test-methodology.md @@ -1,36 +1,20 @@ # Testing a Repo's Workflows -The section below is `WORKFLOW.md` section 5, whole. Which of its checks and scenarios bind a given repository is `WORKFLOW.md` section 1's applicability rule, and the D-guarantees each one answers to are `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. +The section below is `WORKFLOW.md` section 5, whole. Its items and scenarios answer to the D-guarantees in `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. ## The Test Methodology -An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a check or scenario for an absent construct is recorded N/A, not failed. +An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a guarantee or scenario for an absent construct is recorded N/A, not failed. ### 5A. Static Audit (No Execution) -Read the workflow files, `version.json`, and whatever else a check below names as its evidence: a project or dependency file, `global.json`, `codecov.yml`, `.gitignore`, the branch ruleset, and the repo's Actions and Dependabot secret names. Assert the structural fact behind each *applicable* D-guarantee, each pass/fail/N-A with a `file:line` citation, and cite a repository setting by its own name where that setting rather than a file is the evidence. Remember the two layers: assert each input in the layer that declares it. +Assert the structural fact each *applicable* D-guarantee implies, and record **pass**, **fail**, or **N/A** per item. This section says how an audit is run and recorded rather than what must hold: a guarantee names its own constructs, and the requirement is `WORKFLOW.md` section 4's item together with whatever that item defers to. -**Core (every repo):** +Most of the evidence is in the workflow files and the composite actions they reach. Where a guarantee's evidence lies outside them, it is in practice the repo's branch ruleset, its Actions and Dependabot secret names, a workflow the repo only calls, a project or dependency file, or a committed file such as `version.json`, `.github/dependabot.yml`, `global.json`, `codecov.yml`, `.gitignore`, or `.editorconfig`. -- **D1:** a `changes` paths-filter job exists wherever the repo has a smoke build, with one entry per target naming the paths that target is built from, so a change touching no target marks nothing (a filter written as a negation instead marks a docs-only change and fails D1.1); the PR entry workflow's smoke call sets every publish flag its release task declares to false (`github`/`dockerhub`, and a package-push flag there is itself a finding, per `WORKFLOW.md` section 1); a pushing leaf receives `smoke: true` and a derived `push` (false on smoke), and a build-only leaf receives `smoke: true` with no `push` to derive; every `upload-artifact` the smoke call reaches, in a build task and in any job collecting other jobs' artifacts alike, is gated on smoke being false, written `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings; the aggregator runs under `if: always()`, `needs:` the validation job, and the `changes` and `smoke-build` jobs too wherever the repo has a smoke build, blocks on `failure`/`cancelled`, and passes on a **skipped smoke build**, so a no-build repo's aggregator, having only the validation job to read, requires that job to have succeeded; the aggregator's own job `name:` is the string the branch ruleset's required-check `context:` carries; a validation job runs unconditionally. -- **D1.6:** the validator the repo's validation job reaches collects coverage and uploads it, in every C# and Python repo that has tests. Its C# leg runs `dotnet test --coverage --coverage-output-format cobertura --results-directory ./coverage`, with `--coverage-output` unset, wherever the repo's test project is MTP-based. An MTP-based repo also ships a root `global.json` declaring the Microsoft.Testing.Platform runner, references `Microsoft.Testing.Extensions.CodeCoverage` at 18.9.0 or later in place of `coverlet.collector`, and carries no `xunit.runner.visualstudio`, while a repo still on the VSTest collector keeps its existing validator pin, a migration owed rather than drift. Its Python leg runs `pytest --cov-report=xml`, and the repo declares `pytest-cov` among its test dependencies, a dev dependency group in a uv project and a `requirements*.txt` entry on pip, selects the coverage source in its own `pyproject.toml` rather than leaving `--cov` unset, and leaves the report at the repo root as `coverage.xml`. Either way the report reaches a `codecov/codecov-action` step made best-effort by `continue-on-error` and/or `fail_ci_if_error: false`, and `CODECOV_TOKEN` is present in **both** the repo's Actions and its Dependabot secret names, the second because a Dependabot-triggered run reads that store and the upload otherwise skips silently on every bot pull request. `codecov.yml` sets the project and patch statuses `informational: true`, or names the threshold the repo enforces instead, and lists any intentionally-untested, non-shipped project under `ignore`. `.gitignore` excludes the coverage output. Record the whole item N/A for a repo with no tests, and for a type the audited repo carries at the `lint-only` profile. -- **D2:** an entry validation job/step exists per complex-input workflow; the release gate checks both directions, strips `+buildmetadata`, and skips on smoke; the publisher rejects a dispatch from a ref other than `main` or `develop`. -- **D3:** each run builds one branch, so NBGV classifies `github.ref` directly (no `IGNORE_GITHUB_REF`), and the default-branch literal in the gate (`== 'main'`), the `prerelease` expression (`!= 'main'`), and `version.json`'s `publicReleaseRefSpec` all name the repo's actual default branch. `version.json` sets the major.minor floor, and NBGV and `version.json` are both retained even by a repo with no compiler, since they own the tag (D3.3). NuGet.org derives the prerelease flag from the SemVer2 `-g` suffix rather than the workflow setting one, and the PyPI version is built from `AssemblyFileVersion` with `.dev0` appended on `develop` only (D3.4). -- **D4:** `target_commitish` is the NBGV commit id; `prerelease` equals `branch != default`; the release-create step is gated `exists == 'false' || github.event_name == 'workflow_dispatch'` (the step output is the string `'false'`, not a boolean); the asset-delete step carries that same condition, narrowed by `inputs.expect_release_assets`. Where `workflow_dispatch` is the publisher's only trigger (`releaseTrigger: dispatch-only`) every run is a dispatch, so the exists-check's skip leg can never fire: record that leg N/A rather than failed, and expect the release-create step to still carry the `exists == 'false' || github.event_name == 'workflow_dispatch'` condition, since D6.4 keeps the `github-release` job body verbatim. A caller with no file target passes `expect_release_assets: false`, which covers a Docker-only, a PyPI-only, and a source-only repo alike, while a NuGet-only caller keeps the default `true` because its leaf uploads a `release-asset-*` carrying the package, and a source-only caller also sets every `enable_*` input false. `github-release` and the terminal registry pusher (Docker) each carry `!failure() && !cancelled()` rather than the implicit `success()`, so a failed build skips both, while a target that merely skipped, being disabled or unchanged, still lets the release and the Docker push proceed, and a package target's separate publish job `needs:` the release-task call for the same reason (D4.5). A first `plan` job decides once whether the run publishes, admitting a code-affecting bot push to `main`, a dispatch of `main` or `develop`, and a `main`-only schedule, and every publishing job gates on that decision (D4.1). -- **D5:** each cross-job transfer artifact has a delete step at its consumer, gated so it runs exactly when the consumption happened rather than when the whole job succeeded (D5.2), `continue-on-error: true`, tolerating a failed listing, and looping all ids; **every** upload sets `retention-days: 1`; **no** cleanup step enumerates and deletes the run's whole artifact set, in whatever jq or API shape it is written. -- **D6:** the release download uses `pattern:`/`merge-multiple:` (no `artifact-ids:`). Branch-derived config reads `inputs.branch` (a `github.ref_name` in such config is a finding). Artifact names are branch-suffixed. The target set is consistent across every surface D6.4 names: the `enable_` input, the `build-` job, that job's entries in the `github-release` and `build-docker` `needs:` lists, the `changes` paths-filter entry **and** its output, the `smoke-build` enable-forward, and any separate `publish-` job the package-registry seam requires. The `inputs.branch` rule above binds a called leaf, while a `publish-` job is in the publisher and reads `github.ref_name` correctly. -- **D7:** the publisher concurrency group is global and ref-independent with `cancel-in-progress: false`. A reusable job declares `permissions:` only where every caller grants that scope at startup, per D7.2. A boolean used by both `workflow_call` and `workflow_dispatch` is declared in both trigger blocks, and a boolean read through `github.event.inputs.` is compared against `'true'` alone, a comparison against the boolean `true` never firing on a string, while an `inputs.` read carries the declared boolean and is used directly, a both-forms comparison there being redundant rather than a finding, so a repo whose booleans all arrive by `workflow_call` records the comparison half N/A. A job or step output is a string too and takes the same `== 'true'` (D7.3). Every cross-job condition that admits a skipped dependency pairs its allowlist with a status-check function such as `always()` or `!failure() && !cancelled()`, since the implicit `success()` is false the moment a `needs:` job skipped (D7.4). -- **D8/D9:** the merge-bot enables auto-merge on `opened`/`reopened` for every Dependabot tier including semver-major, dispatches `--squash`/`--merge` by the PR's base ref, disables on a maintainer-pushed `synchronize`, and keys concurrency on the PR number rather than `github.ref`. Codegen runs as a matrix over both branches, and Dependabot targets both branches with security PRs to the default branch (D8.2). The upstream tracker's `bump-branch-prefix` and `branches` match a merge-bot rule, unless it sets `auto-merge: false`, which prefixes the head so no rule matches (wrapper repos). A gate comparing `github.actor` against hard-coded bot identities emits a `::warning::` on the non-matching branch, and the annotation is optional only where the failure announces itself anyway (D8.4). Actions are SHA-pinned. Names, shells, and conditionals follow `GOVERNANCE.md` "Workflow YAML Conventions", and line endings follow `.editorconfig` (D9.5), per the policy `GOVERNANCE.md` "Documentation Style Conventions" routes to under "Line Endings". - -**Per-type addenda (apply only the ones present):** - -- **.NET publish:** the smoke runtime set is a strict non-empty subset of the full runtime set. The selected set runs sequentially inside one composite-action job. A non-smoke run uploads one `release-asset--dotnet-publish` artifact, while a smoke run skips the archive and upload steps. -- **NuGet:** `publish-nuget` is a job in the repo's own publisher, never inside the release task and never in a reusable workflow a different repository hosts. `id-token: write` appears on that job only, absent from the build and PR paths, beside `actions: write` for the artifact cleanup. The push uses `--skip-duplicate` and is gated by that job's `needs:` on the release-task call, never on an existence check, so a PR never reaches it. The job consume-then-deletes `nuget-build-` under the download step's own success, per D5.1 and D5.2. `*.nupkg` push also carries the paired `.snupkg` to the symbol server where symbols are enabled. The `release-asset` `.7z` carries the package(s). -- **PyPI:** `publish-pypi` declares `environment: { name: pypi }`. `id-token: write` appears only on that job (absent from the build/PR path). `skip-existing: true` is set on the publish action. The build artifact is deleted under the download step's own success, per D5.2. The `pypi` environment has a deployment-branch rule. -- **Docker:** the leaf reads the external state file for the tag instead of `SemVer2` (wrapper repos only, since a plain Docker repo correctly tags off `SemVer2` and records this N/A). The readme job is gated main-only, both by the caller's branch input and inside the hub-hosted `publish-docker-readme-task.yml` itself. The docker-readme task validates `repositories` XOR `manifest`+`manifest-jq`. The buildcache follows D9.4. -- **Static site deployed to a host:** the generator is pinned by version **and** by a checksum verified before install, declared once across the workflows that install it. The deploy is a dispatch carrying an environment choice, with concurrency keyed on the **environment** and `cancel-in-progress: false`, and production gated to the default branch while any ref may reach a non-production environment. The reusable callee re-asserts the environment name in a job of its own. The upload targets a per-release directory and carries no delete flag at the environment root, and the pointer flip is a separate step. The terminal check asserts the environment, then the release id, then the URL contract, waiting for convergence to a bounded timeout rather than sampling once, and reporting an unreachable host distinctly from an HTTP status (D4.6). Retention is bounded by a declared count and one side is recorded as owning the prune: the deploy asserts it where the credential can observe the destination, and the host owns it where the credential is confined write-only (D5.6). +Cite what each verdict rests on. That is `file:line` for a file in the audited repo, its own name where a setting, a ruleset, or a secret name rather than a file is the evidence, and `/@` plus the `file:line` in that repo where the guarantee binds a workflow or composite action the audited repo only reaches, read at the SHA the caller pins. An **N/A** verdict names the absent construct instead, there being no line to cite. ### 5B. End-to-End Trace Scenarios (No Execution, Deterministic from the YAML) @@ -38,23 +22,23 @@ For each *applicable* scenario, evaluate every job's `if:`/`needs:` against the | # | Input | Expected output | Exercises | | --- | --- | --- | --- | -| S1 | PR touching a build target | `changes` flags it; validation runs; that target's smoke build runs; no push, **no uploads**; validate-release **skipped (smoke), succeeds**; release **skipped**; aggregator **success**; version = prerelease; no release; no dangling artifacts | D1, D2.2, D3 | +| S1 | PR touching a build target | `changes` flags it; validation runs; that target's smoke build runs; no push, **no uploads**; validate-release **succeeds**, its check exiting early on smoke per D2.2; release **skipped**; aggregator **success**; version = prerelease; no release; no dangling artifacts | D1, D2.2, D3 | | S2 | PR changing only docs | smoke-build **skipped**, validation runs, aggregator **success** | D1.1, D1.2, D1.5 | | S3 | PR changing only `.github/workflows/**` | the filter marks no target -> smoke-build **skipped**, validation runs, aggregator **success** | D1.2, D1.4, D1.5 | -| S4 | PR base = default branch, carrying a build target | smoke versions as prerelease, validate-release **skipped (smoke), succeeds** so the default-branch arm does **not** fire, aggregator **success**, promotion not blocked | D1.5, D2.2, D3.2 | +| S4 | PR base = default branch, carrying a build target | smoke versions as prerelease, validate-release **succeeds** with its check exited early per D2.2, so the default-branch arm does **not** fire, aggregator **success**, promotion not blocked | D1.5, D2.2, D3.2 | | S5 | bot push to `main` not touching a release path (e.g. an Actions bump) | the paths filter excludes it, so nothing publishes | D4.1 | | S6 | code-affecting **bot** push to `main` (a human push/promotion, or any develop push, does not) | the `plan` job gates it to the App/Dependabot actor, and `main` publishes a release | D3, D4 | | S7 | publish run (schedule, a bot push to main, or a dispatch) | builds the **one** trigger branch: `main` -> `X.Y.Z`, `prerelease=false`, registry stable, readme run; `develop` -> `X.Y.Z-g`, `prerelease=true`, registry prerelease; `release-asset-*` consumed-then-deleted; each package build-artifact (`nuget-build-*`, `pypi-build-*`) deleted after its publish; **no dangling artifacts** | D3, D4, D5, D6, D7 | | S8 | dispatch from a ref other than `main` or `develop` | **fails fast** | D2.3 | | S9 | re-run publish on a schedule or push trigger, version unchanged (a dispatch re-run refreshes the release instead, per D4.4) | release-create **skipped**, `release-asset-*` delete **skipped**; NuGet/PyPI pushes no-op (server dedupe); **package build-artifacts still deleted** (their download succeeded); **Docker still re-pushes** the image; no duplicate release | D4.4, D5.2 | | S10 | branch/version classification disagree | validate-release **fails loud**, build/publish skip | D2.2 | -| S11 | scheduled upstream-version bump (wrapper) | resolver detects a change -> commits the state file -> opens a `-` PR -> merge-bot auto-merges -> the `main` pin publishes via the gate (a develop pin does not auto-publish, shipping instead via a develop dispatch or promotion) | D8.3, D3.5 | +| S11 | scheduled upstream-version bump (wrapper) | resolver detects a change -> commits the state file -> opens a per-branch bump PR -> the merge-bot auto-merges it, or leaves it for the maintainer where the tracker sets `auto-merge: false` (D8.3) -> the `main` pin publishes via the gate (a develop pin does not auto-publish, shipping instead via a develop dispatch or promotion) | D8.3, D3.5 | | S12 | deploy dispatch naming an environment | the ref gate runs **first** (production from the default branch only, any ref to a non-production environment); validation runs; the callee re-asserts the environment name; a release installs under its own id; the pointer flips as a separate step; retention is bounded by whichever of the two D5.6 shapes the repo uses, so a deploy whose credential can observe the destination asserts the count converged and one confined write-only leaves it to the host; the live check asserts the environment and the release id, waiting out the reload, then the URL contract; **no tag and no release are created** | D2.1, D4.6, D5.6 | | S13 | deploy dispatch of a production environment from a non-default ref | **fails fast**, before anything is installed or written | D2.1 | ### 5C. Live Probe (Where Warranted) -Every probe here that dispatches a workflow or re-runs a real publish is the maintainer's to run, and the agent prepares the command and reads the result back afterwards. A harness that refuses such a write is the harness working as intended, and the refusal is neither re-shaped into a raw API call nor talked around (GOVERNANCE.md "Repository Boundaries and Write Safety"). +Every probe here that opens a pull request, dispatches a workflow, or re-runs a real publish is the maintainer's to run, with the agent preparing the command and reading the result back afterwards. A harness that refuses such a write is the harness working as intended, and the refusal is neither re-shaped into a raw API call nor talked around (`GOVERNANCE.md` "Repository Boundaries and Write Safety"). - Open a trivial-change PR touching one target and confirm S1. *Caveat: the Docker leg logs in to the registry even on smoke and reads the buildcache, so it needs `DOCKER_HUB_*` secrets and cannot run on a fork PR (same-repo only).* - Per registry: after a real publish, query NuGet.org for the expected version + prerelease classification (and the `.snupkg` on the symbol server), and confirm a re-run added no duplicate. For PyPI read the built `dist/*` filenames out of the build job's log, `.dev0` off `develop` vs a plain version on the default branch. @@ -65,9 +49,9 @@ Every probe here that dispatches a workflow or re-runs a real publish is the mai Record the workflow **operational** when every *applicable* 5A item passes, every *applicable* 5B scenario's predicted output equals the expected, and no 5C probe that was run contradicts either. N/A items are excluded, never counted as failures. Any *applicable* mismatch is a **defect** -> **not operational**. Procedure: -1. **Audit** with 5A. Record pass/fail/N-A with `file:line`. +1. **Audit** with 5A, recording each item's verdict and its evidence in the form 5A sets out. 2. **Trace** the applicable S-scenarios with 5B. Diff predicted vs expected. -3. **Probe** with 5C where a live signal exists that the static trace cannot produce: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. +3. **Probe** with 5C where a live signal exists that the static trace cannot produce, running the probes that only read and preparing the writing ones for the maintainer: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. 4. **Verdict:** operational / not operational, with the failing guarantee(s) and the triggering input for each, the list of items recorded N/A, and the 5C probes prepared but not run. `WORKFLOW.md` section 5 keeps the test methodology, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/test-methodology.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. diff --git a/.github/skills/audit-a-repo/SKILL.md b/.github/skills/audit-a-repo/SKILL.md index 5cd0c133..070703f9 100644 --- a/.github/skills/audit-a-repo/SKILL.md +++ b/.github/skills/audit-a-repo/SKILL.md @@ -18,10 +18,10 @@ The audit is the fleet's measurement procedure, and the two failure shapes it gu ## Measuring -- **Resolve the repo's types from `registry/repos.json`** and classify a `classificationPending` entry from the tree (`AUDIT.md` section 2). The applicability gate is `WORKFLOW.md` section 1: a check governing an absent construct is N/A, excluded from the verdict, and never a defect (`AUDIT.md` section 3). +- **Resolve the repo's types from `registry/repos.json`** and classify a `classificationPending` entry from the tree (`AUDIT.md` section 2). The applicability gate is `WORKFLOW.md` section 1, extended to `AUDIT.md`'s own checks: an item or check governing an absent construct is N/A, excluded from the verdict, and never a defect (`AUDIT.md` section 3). - **Know what the runner does and does not prove.** `spec/audit.py` mechanizes the deterministic subset only: settings, rulesets, secret names, file and section presence, verbatim hashing, interface wiring, Dependabot coverage, branch facts. It evaluates no check under a type in `spec/project-types.json`, so every per-type check is judged by hand, and a clean run is no evidence for them (`AUDIT.md` section 4). Silence from a tool that was never looking reads exactly like a pass. - **Judge letter and intent per check** and keep the vocabulary: letter miss with intent satisfied is a drift finding, both missing is a defect, and operational is binary over the applicable set (`AUDIT.md` sections 4 and 7). Do not invent a parallel scheme. -- **Assert the Actions implement `WORKFLOW.md`** by outcome, not by matching catalog snippets byte for byte: the 5A static audit with a `file:line` citation per applicable guarantee, then the 5B trace scenarios (`AUDIT.md` section 5). The `workflow-ci-contract` skill summarizes that contract. +- **Assert the Actions implement `WORKFLOW.md`** by outcome, not by matching catalog snippets byte for byte: the 5A static audit, each applicable guarantee cited in the form 5A sets out, then the 5B trace scenarios (`AUDIT.md` section 5). Read a workflow the repo only calls at the SHA it pins, for both. The `workflow-ci-contract` skill summarizes that contract. - **Check live settings, rulesets, and secrets from a hub checkout at `main`** with `AUDIT.md` section 6. Run `repo-config/configure.sh check` with the target repository and model for settings and rulesets, and `spec/audit.py [RepoName]` for secrets, rather than constructing a local comparison. The hub payloads are the only repository-configuration source. ## Reporting diff --git a/.github/skills/workflow-ci-contract/SKILL.md b/.github/skills/workflow-ci-contract/SKILL.md index 8f751413..a6fe9ab2 100644 --- a/.github/skills/workflow-ci-contract/SKILL.md +++ b/.github/skills/workflow-ci-contract/SKILL.md @@ -13,10 +13,10 @@ description: >- ## How the Contract Is Read - **Outcomes, not bytes.** A workflow is judged against `WORKFLOW.md` section 4's expected inputs and outputs, never against a snippet byte for byte, per `GOVERNANCE.md` "Foundational Principles". -- **Applicability.** A guarantee, or a 5A check or 5B scenario from `WORKFLOW.md` section 5, governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. +- **Applicability.** A guarantee, or a 5B scenario from `WORKFLOW.md` section 5, governing a construct the repo does not contain is N/A: recorded, excluded from the verdict, never a defect. A source-only pipeline is mostly N/A and that is fine. - **Operational is binary.** Every applicable guarantee holds, or the workflow is not operational. A single applicable input-output mismatch is a defect regardless of how clean the YAML looks. - **Reached, not carried.** A standard workflow whose job graph is identical across repos of a type is reached as a hub-hosted `workflow_call` task, per `GOVERNANCE.md` "Hub-Hosted Tooling". The repo's own surface is the caller stub, pinned to a hub release commit, and a composite-action hook at `.github/actions/` for what is its own. A hub task reaches its own actions and sibling tasks through `$/`, which resolves at that pinned commit. The merge-bot is the first, and `docs/reusable-workflows.md` in the hub carries the model, the hook contract, and the stage each workflow migrates in. Until a workflow's stage ships, its copy is graded against the same contract. -- **Two layers.** The pipeline splits into an orchestrator layer and a build-leaf layer, defined in `WORKFLOW.md` section 3's `Two Layers: Orchestration vs Build` and carried in `references/architecture.md`, while `WORKFLOW.md` section 1's `Two layers when auditing` maps which layer declares which input. When a check names an input, assert it in the layer that declares it. +- **Two layers.** The pipeline splits into an orchestrator layer and a build-leaf layer, defined in `WORKFLOW.md` section 3's `Two Layers: Orchestration vs Build` and carried in `references/architecture.md`, while `WORKFLOW.md` section 1's `Two layers when auditing` maps which layer declares which input. Assert an input a guarantee names in the layer that declares it. ## Style Rules diff --git a/.github/skills/workflow-ci-contract/references/test-methodology.md b/.github/skills/workflow-ci-contract/references/test-methodology.md index 744bff09..1dd10c14 100644 --- a/.github/skills/workflow-ci-contract/references/test-methodology.md +++ b/.github/skills/workflow-ci-contract/references/test-methodology.md @@ -1,36 +1,20 @@ # Testing a Repo's Workflows -The section below is `WORKFLOW.md` section 5, whole. Which of its checks and scenarios bind a given repository is `WORKFLOW.md` section 1's applicability rule, and the D-guarantees each one answers to are `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. +The section below is `WORKFLOW.md` section 5, whole. Its items and scenarios answer to the D-guarantees in `WORKFLOW.md` section 4, carried whole in `d-guarantees.md` beside this file. ## The Test Methodology -An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a check or scenario for an absent construct is recorded N/A, not failed. +An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a guarantee or scenario for an absent construct is recorded N/A, not failed. ### 5A. Static Audit (No Execution) -Read the workflow files, `version.json`, and whatever else a check below names as its evidence: a project or dependency file, `global.json`, `codecov.yml`, `.gitignore`, the branch ruleset, and the repo's Actions and Dependabot secret names. Assert the structural fact behind each *applicable* D-guarantee, each pass/fail/N-A with a `file:line` citation, and cite a repository setting by its own name where that setting rather than a file is the evidence. Remember the two layers: assert each input in the layer that declares it. +Assert the structural fact each *applicable* D-guarantee implies, and record **pass**, **fail**, or **N/A** per item. This section says how an audit is run and recorded rather than what must hold: a guarantee names its own constructs, and the requirement is `WORKFLOW.md` section 4's item together with whatever that item defers to. -**Core (every repo):** +Most of the evidence is in the workflow files and the composite actions they reach. Where a guarantee's evidence lies outside them, it is in practice the repo's branch ruleset, its Actions and Dependabot secret names, a workflow the repo only calls, a project or dependency file, or a committed file such as `version.json`, `.github/dependabot.yml`, `global.json`, `codecov.yml`, `.gitignore`, or `.editorconfig`. -- **D1:** a `changes` paths-filter job exists wherever the repo has a smoke build, with one entry per target naming the paths that target is built from, so a change touching no target marks nothing (a filter written as a negation instead marks a docs-only change and fails D1.1); the PR entry workflow's smoke call sets every publish flag its release task declares to false (`github`/`dockerhub`, and a package-push flag there is itself a finding, per `WORKFLOW.md` section 1); a pushing leaf receives `smoke: true` and a derived `push` (false on smoke), and a build-only leaf receives `smoke: true` with no `push` to derive; every `upload-artifact` the smoke call reaches, in a build task and in any job collecting other jobs' artifacts alike, is gated on smoke being false, written `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings; the aggregator runs under `if: always()`, `needs:` the validation job, and the `changes` and `smoke-build` jobs too wherever the repo has a smoke build, blocks on `failure`/`cancelled`, and passes on a **skipped smoke build**, so a no-build repo's aggregator, having only the validation job to read, requires that job to have succeeded; the aggregator's own job `name:` is the string the branch ruleset's required-check `context:` carries; a validation job runs unconditionally. -- **D1.6:** the validator the repo's validation job reaches collects coverage and uploads it, in every C# and Python repo that has tests. Its C# leg runs `dotnet test --coverage --coverage-output-format cobertura --results-directory ./coverage`, with `--coverage-output` unset, wherever the repo's test project is MTP-based. An MTP-based repo also ships a root `global.json` declaring the Microsoft.Testing.Platform runner, references `Microsoft.Testing.Extensions.CodeCoverage` at 18.9.0 or later in place of `coverlet.collector`, and carries no `xunit.runner.visualstudio`, while a repo still on the VSTest collector keeps its existing validator pin, a migration owed rather than drift. Its Python leg runs `pytest --cov-report=xml`, and the repo declares `pytest-cov` among its test dependencies, a dev dependency group in a uv project and a `requirements*.txt` entry on pip, selects the coverage source in its own `pyproject.toml` rather than leaving `--cov` unset, and leaves the report at the repo root as `coverage.xml`. Either way the report reaches a `codecov/codecov-action` step made best-effort by `continue-on-error` and/or `fail_ci_if_error: false`, and `CODECOV_TOKEN` is present in **both** the repo's Actions and its Dependabot secret names, the second because a Dependabot-triggered run reads that store and the upload otherwise skips silently on every bot pull request. `codecov.yml` sets the project and patch statuses `informational: true`, or names the threshold the repo enforces instead, and lists any intentionally-untested, non-shipped project under `ignore`. `.gitignore` excludes the coverage output. Record the whole item N/A for a repo with no tests, and for a type the audited repo carries at the `lint-only` profile. -- **D2:** an entry validation job/step exists per complex-input workflow; the release gate checks both directions, strips `+buildmetadata`, and skips on smoke; the publisher rejects a dispatch from a ref other than `main` or `develop`. -- **D3:** each run builds one branch, so NBGV classifies `github.ref` directly (no `IGNORE_GITHUB_REF`), and the default-branch literal in the gate (`== 'main'`), the `prerelease` expression (`!= 'main'`), and `version.json`'s `publicReleaseRefSpec` all name the repo's actual default branch. `version.json` sets the major.minor floor, and NBGV and `version.json` are both retained even by a repo with no compiler, since they own the tag (D3.3). NuGet.org derives the prerelease flag from the SemVer2 `-g` suffix rather than the workflow setting one, and the PyPI version is built from `AssemblyFileVersion` with `.dev0` appended on `develop` only (D3.4). -- **D4:** `target_commitish` is the NBGV commit id; `prerelease` equals `branch != default`; the release-create step is gated `exists == 'false' || github.event_name == 'workflow_dispatch'` (the step output is the string `'false'`, not a boolean); the asset-delete step carries that same condition, narrowed by `inputs.expect_release_assets`. Where `workflow_dispatch` is the publisher's only trigger (`releaseTrigger: dispatch-only`) every run is a dispatch, so the exists-check's skip leg can never fire: record that leg N/A rather than failed, and expect the release-create step to still carry the `exists == 'false' || github.event_name == 'workflow_dispatch'` condition, since D6.4 keeps the `github-release` job body verbatim. A caller with no file target passes `expect_release_assets: false`, which covers a Docker-only, a PyPI-only, and a source-only repo alike, while a NuGet-only caller keeps the default `true` because its leaf uploads a `release-asset-*` carrying the package, and a source-only caller also sets every `enable_*` input false. `github-release` and the terminal registry pusher (Docker) each carry `!failure() && !cancelled()` rather than the implicit `success()`, so a failed build skips both, while a target that merely skipped, being disabled or unchanged, still lets the release and the Docker push proceed, and a package target's separate publish job `needs:` the release-task call for the same reason (D4.5). A first `plan` job decides once whether the run publishes, admitting a code-affecting bot push to `main`, a dispatch of `main` or `develop`, and a `main`-only schedule, and every publishing job gates on that decision (D4.1). -- **D5:** each cross-job transfer artifact has a delete step at its consumer, gated so it runs exactly when the consumption happened rather than when the whole job succeeded (D5.2), `continue-on-error: true`, tolerating a failed listing, and looping all ids; **every** upload sets `retention-days: 1`; **no** cleanup step enumerates and deletes the run's whole artifact set, in whatever jq or API shape it is written. -- **D6:** the release download uses `pattern:`/`merge-multiple:` (no `artifact-ids:`). Branch-derived config reads `inputs.branch` (a `github.ref_name` in such config is a finding). Artifact names are branch-suffixed. The target set is consistent across every surface D6.4 names: the `enable_` input, the `build-` job, that job's entries in the `github-release` and `build-docker` `needs:` lists, the `changes` paths-filter entry **and** its output, the `smoke-build` enable-forward, and any separate `publish-` job the package-registry seam requires. The `inputs.branch` rule above binds a called leaf, while a `publish-` job is in the publisher and reads `github.ref_name` correctly. -- **D7:** the publisher concurrency group is global and ref-independent with `cancel-in-progress: false`. A reusable job declares `permissions:` only where every caller grants that scope at startup, per D7.2. A boolean used by both `workflow_call` and `workflow_dispatch` is declared in both trigger blocks, and a boolean read through `github.event.inputs.` is compared against `'true'` alone, a comparison against the boolean `true` never firing on a string, while an `inputs.` read carries the declared boolean and is used directly, a both-forms comparison there being redundant rather than a finding, so a repo whose booleans all arrive by `workflow_call` records the comparison half N/A. A job or step output is a string too and takes the same `== 'true'` (D7.3). Every cross-job condition that admits a skipped dependency pairs its allowlist with a status-check function such as `always()` or `!failure() && !cancelled()`, since the implicit `success()` is false the moment a `needs:` job skipped (D7.4). -- **D8/D9:** the merge-bot enables auto-merge on `opened`/`reopened` for every Dependabot tier including semver-major, dispatches `--squash`/`--merge` by the PR's base ref, disables on a maintainer-pushed `synchronize`, and keys concurrency on the PR number rather than `github.ref`. Codegen runs as a matrix over both branches, and Dependabot targets both branches with security PRs to the default branch (D8.2). The upstream tracker's `bump-branch-prefix` and `branches` match a merge-bot rule, unless it sets `auto-merge: false`, which prefixes the head so no rule matches (wrapper repos). A gate comparing `github.actor` against hard-coded bot identities emits a `::warning::` on the non-matching branch, and the annotation is optional only where the failure announces itself anyway (D8.4). Actions are SHA-pinned. Names, shells, and conditionals follow `GOVERNANCE.md` "Workflow YAML Conventions", and line endings follow `.editorconfig` (D9.5), per the policy `GOVERNANCE.md` "Documentation Style Conventions" routes to under "Line Endings". - -**Per-type addenda (apply only the ones present):** - -- **.NET publish:** the smoke runtime set is a strict non-empty subset of the full runtime set. The selected set runs sequentially inside one composite-action job. A non-smoke run uploads one `release-asset--dotnet-publish` artifact, while a smoke run skips the archive and upload steps. -- **NuGet:** `publish-nuget` is a job in the repo's own publisher, never inside the release task and never in a reusable workflow a different repository hosts. `id-token: write` appears on that job only, absent from the build and PR paths, beside `actions: write` for the artifact cleanup. The push uses `--skip-duplicate` and is gated by that job's `needs:` on the release-task call, never on an existence check, so a PR never reaches it. The job consume-then-deletes `nuget-build-` under the download step's own success, per D5.1 and D5.2. `*.nupkg` push also carries the paired `.snupkg` to the symbol server where symbols are enabled. The `release-asset` `.7z` carries the package(s). -- **PyPI:** `publish-pypi` declares `environment: { name: pypi }`. `id-token: write` appears only on that job (absent from the build/PR path). `skip-existing: true` is set on the publish action. The build artifact is deleted under the download step's own success, per D5.2. The `pypi` environment has a deployment-branch rule. -- **Docker:** the leaf reads the external state file for the tag instead of `SemVer2` (wrapper repos only, since a plain Docker repo correctly tags off `SemVer2` and records this N/A). The readme job is gated main-only, both by the caller's branch input and inside the hub-hosted `publish-docker-readme-task.yml` itself. The docker-readme task validates `repositories` XOR `manifest`+`manifest-jq`. The buildcache follows D9.4. -- **Static site deployed to a host:** the generator is pinned by version **and** by a checksum verified before install, declared once across the workflows that install it. The deploy is a dispatch carrying an environment choice, with concurrency keyed on the **environment** and `cancel-in-progress: false`, and production gated to the default branch while any ref may reach a non-production environment. The reusable callee re-asserts the environment name in a job of its own. The upload targets a per-release directory and carries no delete flag at the environment root, and the pointer flip is a separate step. The terminal check asserts the environment, then the release id, then the URL contract, waiting for convergence to a bounded timeout rather than sampling once, and reporting an unreachable host distinctly from an HTTP status (D4.6). Retention is bounded by a declared count and one side is recorded as owning the prune: the deploy asserts it where the credential can observe the destination, and the host owns it where the credential is confined write-only (D5.6). +Cite what each verdict rests on. That is `file:line` for a file in the audited repo, its own name where a setting, a ruleset, or a secret name rather than a file is the evidence, and `/@` plus the `file:line` in that repo where the guarantee binds a workflow or composite action the audited repo only reaches, read at the SHA the caller pins. An **N/A** verdict names the absent construct instead, there being no line to cite. ### 5B. End-to-End Trace Scenarios (No Execution, Deterministic from the YAML) @@ -38,23 +22,23 @@ For each *applicable* scenario, evaluate every job's `if:`/`needs:` against the | # | Input | Expected output | Exercises | | --- | --- | --- | --- | -| S1 | PR touching a build target | `changes` flags it; validation runs; that target's smoke build runs; no push, **no uploads**; validate-release **skipped (smoke), succeeds**; release **skipped**; aggregator **success**; version = prerelease; no release; no dangling artifacts | D1, D2.2, D3 | +| S1 | PR touching a build target | `changes` flags it; validation runs; that target's smoke build runs; no push, **no uploads**; validate-release **succeeds**, its check exiting early on smoke per D2.2; release **skipped**; aggregator **success**; version = prerelease; no release; no dangling artifacts | D1, D2.2, D3 | | S2 | PR changing only docs | smoke-build **skipped**, validation runs, aggregator **success** | D1.1, D1.2, D1.5 | | S3 | PR changing only `.github/workflows/**` | the filter marks no target -> smoke-build **skipped**, validation runs, aggregator **success** | D1.2, D1.4, D1.5 | -| S4 | PR base = default branch, carrying a build target | smoke versions as prerelease, validate-release **skipped (smoke), succeeds** so the default-branch arm does **not** fire, aggregator **success**, promotion not blocked | D1.5, D2.2, D3.2 | +| S4 | PR base = default branch, carrying a build target | smoke versions as prerelease, validate-release **succeeds** with its check exited early per D2.2, so the default-branch arm does **not** fire, aggregator **success**, promotion not blocked | D1.5, D2.2, D3.2 | | S5 | bot push to `main` not touching a release path (e.g. an Actions bump) | the paths filter excludes it, so nothing publishes | D4.1 | | S6 | code-affecting **bot** push to `main` (a human push/promotion, or any develop push, does not) | the `plan` job gates it to the App/Dependabot actor, and `main` publishes a release | D3, D4 | | S7 | publish run (schedule, a bot push to main, or a dispatch) | builds the **one** trigger branch: `main` -> `X.Y.Z`, `prerelease=false`, registry stable, readme run; `develop` -> `X.Y.Z-g`, `prerelease=true`, registry prerelease; `release-asset-*` consumed-then-deleted; each package build-artifact (`nuget-build-*`, `pypi-build-*`) deleted after its publish; **no dangling artifacts** | D3, D4, D5, D6, D7 | | S8 | dispatch from a ref other than `main` or `develop` | **fails fast** | D2.3 | | S9 | re-run publish on a schedule or push trigger, version unchanged (a dispatch re-run refreshes the release instead, per D4.4) | release-create **skipped**, `release-asset-*` delete **skipped**; NuGet/PyPI pushes no-op (server dedupe); **package build-artifacts still deleted** (their download succeeded); **Docker still re-pushes** the image; no duplicate release | D4.4, D5.2 | | S10 | branch/version classification disagree | validate-release **fails loud**, build/publish skip | D2.2 | -| S11 | scheduled upstream-version bump (wrapper) | resolver detects a change -> commits the state file -> opens a `-` PR -> merge-bot auto-merges -> the `main` pin publishes via the gate (a develop pin does not auto-publish, shipping instead via a develop dispatch or promotion) | D8.3, D3.5 | +| S11 | scheduled upstream-version bump (wrapper) | resolver detects a change -> commits the state file -> opens a per-branch bump PR -> the merge-bot auto-merges it, or leaves it for the maintainer where the tracker sets `auto-merge: false` (D8.3) -> the `main` pin publishes via the gate (a develop pin does not auto-publish, shipping instead via a develop dispatch or promotion) | D8.3, D3.5 | | S12 | deploy dispatch naming an environment | the ref gate runs **first** (production from the default branch only, any ref to a non-production environment); validation runs; the callee re-asserts the environment name; a release installs under its own id; the pointer flips as a separate step; retention is bounded by whichever of the two D5.6 shapes the repo uses, so a deploy whose credential can observe the destination asserts the count converged and one confined write-only leaves it to the host; the live check asserts the environment and the release id, waiting out the reload, then the URL contract; **no tag and no release are created** | D2.1, D4.6, D5.6 | | S13 | deploy dispatch of a production environment from a non-default ref | **fails fast**, before anything is installed or written | D2.1 | ### 5C. Live Probe (Where Warranted) -Every probe here that dispatches a workflow or re-runs a real publish is the maintainer's to run, and the agent prepares the command and reads the result back afterwards. A harness that refuses such a write is the harness working as intended, and the refusal is neither re-shaped into a raw API call nor talked around (GOVERNANCE.md "Repository Boundaries and Write Safety"). +Every probe here that opens a pull request, dispatches a workflow, or re-runs a real publish is the maintainer's to run, with the agent preparing the command and reading the result back afterwards. A harness that refuses such a write is the harness working as intended, and the refusal is neither re-shaped into a raw API call nor talked around (`GOVERNANCE.md` "Repository Boundaries and Write Safety"). - Open a trivial-change PR touching one target and confirm S1. *Caveat: the Docker leg logs in to the registry even on smoke and reads the buildcache, so it needs `DOCKER_HUB_*` secrets and cannot run on a fork PR (same-repo only).* - Per registry: after a real publish, query NuGet.org for the expected version + prerelease classification (and the `.snupkg` on the symbol server), and confirm a re-run added no duplicate. For PyPI read the built `dist/*` filenames out of the build job's log, `.dev0` off `develop` vs a plain version on the default branch. @@ -65,9 +49,9 @@ Every probe here that dispatches a workflow or re-runs a real publish is the mai Record the workflow **operational** when every *applicable* 5A item passes, every *applicable* 5B scenario's predicted output equals the expected, and no 5C probe that was run contradicts either. N/A items are excluded, never counted as failures. Any *applicable* mismatch is a **defect** -> **not operational**. Procedure: -1. **Audit** with 5A. Record pass/fail/N-A with `file:line`. +1. **Audit** with 5A, recording each item's verdict and its evidence in the form 5A sets out. 2. **Trace** the applicable S-scenarios with 5B. Diff predicted vs expected. -3. **Probe** with 5C where a live signal exists that the static trace cannot produce: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. +3. **Probe** with 5C where a live signal exists that the static trace cannot produce, running the probes that only read and preparing the writing ones for the maintainer: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. 4. **Verdict:** operational / not operational, with the failing guarantee(s) and the triggering input for each, the list of items recorded N/A, and the 5C probes prepared but not run. `WORKFLOW.md` section 5 keeps the test methodology, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/test-methodology.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. diff --git a/AUDIT.md b/AUDIT.md index 190a6b95..22b2f40e 100644 --- a/AUDIT.md +++ b/AUDIT.md @@ -12,7 +12,7 @@ flowchart TD s0m["0m: fleet membership, every owned non-fork repo has a registry entry"] --> s0["0: has the repo been stood up? if not, STANDUP.md"] s0 --> s1["1: scope, ground-truth branch (main)"] s1 --> s2["2: resolve the repo's type(s)"] - s2 --> s3["3: applicability gate, per check"] + s2 --> s3["3: applicability gate, per item or check"] s3 --> s4["4: per-dimension checks, letter and intent"] s4 --> s5["5: assert Actions implement WORKFLOW.md"] s5 --> s6["6: validate settings, rulesets, secrets"] @@ -61,7 +61,7 @@ Otherwise read its `types[]`. If the entry is `classificationPending` (a backlog ## 3. Applicability Gate -Reuse [`WORKFLOW.md`][workflow] section 1: a check that governs a construct the repo does not contain is **N/A**. Record it as N/A and **exclude it from the verdict**. N/A is never a defect. A Docker check on a repo with no image, a NuGet check on a Python package, and the artifact-lifecycle clauses on a source-only repo are all N/A. +Reuse [`WORKFLOW.md`][workflow] section 1, extended to this audit's own checks: an item or check that governs a construct the repo does not contain is **N/A**. Record it as N/A and **exclude it from the verdict**. N/A is never a defect. A Docker check on a repo with no image, a NuGet check on a Python package, and the artifact-lifecycle clauses on a source-only repo are all N/A. Which carried files and sections a repo is expected to have is decided by its scope selectors (its type(s) plus workflow model, release trigger, and consumer model). The scope model and the `appliesTo` selector vocabulary are defined in [`spec/scope-model.md`][scope-model]. @@ -94,7 +94,7 @@ A check with `intentRef`/`workflowRef` points at the prose section that owns the ## 5. Assert the Actions Implement WORKFLOW.md -Run [`WORKFLOW.md`][workflow]'s methodology against the repo's **own** Actions: the 5A static audit (structural facts per applicable D-guarantee, each with a `file:line` citation) and the 5B trace scenarios (predicted run/skip + version + release + artifact-end-state vs expected). The contract in WORKFLOW.md section 4 is satisfied by **outcome**, not by matching the catalog snippets in [`catalog/snippets/workflows/`][workflows] byte for byte. Those are the reference implementation, not required bytes. +Run [`WORKFLOW.md`][workflow]'s methodology against the repo's **own** Actions, reading a workflow it only calls at the SHA it pins: the 5A static audit (structural facts per applicable D-guarantee, each cited in the form 5A sets out) and the 5B trace scenarios (predicted run/skip + version + release + artifact-end-state vs expected). The contract in WORKFLOW.md section 4 is satisfied by **outcome**, not by matching the catalog snippets in [`catalog/snippets/workflows/`][workflows] byte for byte. Those are the reference implementation, not required bytes. ## 6. Validate Settings, Rulesets, and Secrets diff --git a/WORKFLOW.md b/WORKFLOW.md index 197dff1c..0d4617f5 100644 --- a/WORKFLOW.md +++ b/WORKFLOW.md @@ -17,10 +17,10 @@ The guarantees are distilled from failures observed in practice. Section 4's pre ## 1. Purpose and How to Use This Document - **Contract, not implementation.** Conform to the *outcomes* in section 4. Shape, job names, and file layout may differ between repos, but the input/output behavior may not. -- **Applicability.** A guarantee (or a 5A check, or a 5B scenario) is **applicable** only if the repo contains the construct it governs: a given target, a transfer artifact, a registry push, a wrapper-version source. An item that governs an absent construct is **N/A**: record it as N/A and **exclude it from the verdict**. N/A is never a defect. Section 6 names which items go N/A per project type. A near-empty pipeline (source-only) is mostly N/A and that is fine. +- **Applicability.** A guarantee, or a 5B scenario, is **applicable** only if the repo contains the construct it governs: a given target, a transfer artifact, a registry push, a wrapper-version source. An item that governs an absent construct is **N/A**: record it as N/A and **exclude it from the verdict**. N/A is never a defect. Section 6 names which items go N/A per project type. A near-empty pipeline (source-only) is mostly N/A and that is fine. - **Operational is binary.** A workflow is operational only if every *applicable* guarantee holds. A single applicable input/output mismatch is a defect and makes the workflow non-operational, regardless of how clean the YAML looks. -- **Default branch.** Guarantees say "default branch" portably. It is implemented as the literal `main` in several places (the validate gate, the `prerelease` expression, and `version.json`'s `publicReleaseRefSpec`). These MUST all reference the repo's *actual* default branch. A divergence is a defect (section 5A). -- **Two layers when auditing.** The pipeline splits into an **orchestrator** layer (the PR entry workflow, the publisher, and the version and release jobs) and a **build-leaf** layer (the `build-` tasks, whether separate files or jobs inside the release task). Inputs like `github`/`dockerhub`/`expect_release_assets` live on the orchestrator. A leaf receives `ref`/`branch`/`smoke` and whatever else its own target needs, a derived `push` among them where that leaf pushes. A package target declares no push input on either layer, because section 3's `Output Seam by Destination` puts its push in a `publish-` job in the publisher, gated by `needs:` rather than by a flag. When a check names an input, assert it in the layer that declares it. +- **Default branch.** Guarantees say "default branch" portably. It is implemented as the literal `main` in several places (the validate gate, the `prerelease` expression, and `version.json`'s `publicReleaseRefSpec`). These MUST all reference the repo's *actual* default branch. A divergence is a defect (D3.2). +- **Two layers when auditing.** The pipeline splits into an **orchestrator** layer (the PR entry workflow, the publisher, and the version and release jobs) and a **build-leaf** layer (the `build-` tasks, whether separate files or jobs inside the release task). Inputs like `github`/`dockerhub`/`expect_release_assets` live on the orchestrator. A leaf receives `ref`/`branch`/`smoke` and whatever else its own target needs, a derived `push` among them where that leaf pushes. A package target declares no push input on either layer, because section 3's `Output Seam by Destination` puts its push in a `publish-` job in the publisher, gated by `needs:` rather than by a flag. Assert an input a guarantee names in the layer that declares it. - **The three verbs.** Audit (static), Test (trace + probe), Assess (verdict). Section 5 gives the exact procedure. ## 2. Workflow Style Conventions @@ -213,31 +213,15 @@ The required behaviors, organized by domain. Each is a **MUST**, stated as the o ## 5. Test Methodology -An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a check or scenario for an absent construct is recorded N/A, not failed. +An agent verifies a project in three escalating modes, then renders a verdict. **Skip N/A items** (`WORKFLOW.md` section 1): a guarantee or scenario for an absent construct is recorded N/A, not failed. ### 5A. Static Audit (No Execution) -Read the workflow files, `version.json`, and whatever else a check below names as its evidence: a project or dependency file, `global.json`, `codecov.yml`, `.gitignore`, the branch ruleset, and the repo's Actions and Dependabot secret names. Assert the structural fact behind each *applicable* D-guarantee, each pass/fail/N-A with a `file:line` citation, and cite a repository setting by its own name where that setting rather than a file is the evidence. Remember the two layers: assert each input in the layer that declares it. +Assert the structural fact each *applicable* D-guarantee implies, and record **pass**, **fail**, or **N/A** per item. This section says how an audit is run and recorded rather than what must hold: a guarantee names its own constructs, and the requirement is `WORKFLOW.md` section 4's item together with whatever that item defers to. -**Core (every repo):** +Most of the evidence is in the workflow files and the composite actions they reach. Where a guarantee's evidence lies outside them, it is in practice the repo's branch ruleset, its Actions and Dependabot secret names, a workflow the repo only calls, a project or dependency file, or a committed file such as `version.json`, `.github/dependabot.yml`, `global.json`, `codecov.yml`, `.gitignore`, or `.editorconfig`. -- **D1:** a `changes` paths-filter job exists wherever the repo has a smoke build, with one entry per target naming the paths that target is built from, so a change touching no target marks nothing (a filter written as a negation instead marks a docs-only change and fails D1.1); the PR entry workflow's smoke call sets every publish flag its release task declares to false (`github`/`dockerhub`, and a package-push flag there is itself a finding, per `WORKFLOW.md` section 1); a pushing leaf receives `smoke: true` and a derived `push` (false on smoke), and a build-only leaf receives `smoke: true` with no `push` to derive; every `upload-artifact` the smoke call reaches, in a build task and in any job collecting other jobs' artifacts alike, is gated on smoke being false, written `!inputs.smoke` at the workflow layer and `inputs.smoke != 'true'` in a composite action, whose inputs are strings; the aggregator runs under `if: always()`, `needs:` the validation job, and the `changes` and `smoke-build` jobs too wherever the repo has a smoke build, blocks on `failure`/`cancelled`, and passes on a **skipped smoke build**, so a no-build repo's aggregator, having only the validation job to read, requires that job to have succeeded; the aggregator's own job `name:` is the string the branch ruleset's required-check `context:` carries; a validation job runs unconditionally. -- **D1.6:** the validator the repo's validation job reaches collects coverage and uploads it, in every C# and Python repo that has tests. Its C# leg runs `dotnet test --coverage --coverage-output-format cobertura --results-directory ./coverage`, with `--coverage-output` unset, wherever the repo's test project is MTP-based. An MTP-based repo also ships a root `global.json` declaring the Microsoft.Testing.Platform runner, references `Microsoft.Testing.Extensions.CodeCoverage` at 18.9.0 or later in place of `coverlet.collector`, and carries no `xunit.runner.visualstudio`, while a repo still on the VSTest collector keeps its existing validator pin, a migration owed rather than drift. Its Python leg runs `pytest --cov-report=xml`, and the repo declares `pytest-cov` among its test dependencies, a dev dependency group in a uv project and a `requirements*.txt` entry on pip, selects the coverage source in its own `pyproject.toml` rather than leaving `--cov` unset, and leaves the report at the repo root as `coverage.xml`. Either way the report reaches a `codecov/codecov-action` step made best-effort by `continue-on-error` and/or `fail_ci_if_error: false`, and `CODECOV_TOKEN` is present in **both** the repo's Actions and its Dependabot secret names, the second because a Dependabot-triggered run reads that store and the upload otherwise skips silently on every bot pull request. `codecov.yml` sets the project and patch statuses `informational: true`, or names the threshold the repo enforces instead, and lists any intentionally-untested, non-shipped project under `ignore`. `.gitignore` excludes the coverage output. Record the whole item N/A for a repo with no tests, and for a type the audited repo carries at the `lint-only` profile. -- **D2:** an entry validation job/step exists per complex-input workflow; the release gate checks both directions, strips `+buildmetadata`, and skips on smoke; the publisher rejects a dispatch from a ref other than `main` or `develop`. -- **D3:** each run builds one branch, so NBGV classifies `github.ref` directly (no `IGNORE_GITHUB_REF`), and the default-branch literal in the gate (`== 'main'`), the `prerelease` expression (`!= 'main'`), and `version.json`'s `publicReleaseRefSpec` all name the repo's actual default branch. `version.json` sets the major.minor floor, and NBGV and `version.json` are both retained even by a repo with no compiler, since they own the tag (D3.3). NuGet.org derives the prerelease flag from the SemVer2 `-g` suffix rather than the workflow setting one, and the PyPI version is built from `AssemblyFileVersion` with `.dev0` appended on `develop` only (D3.4). -- **D4:** `target_commitish` is the NBGV commit id; `prerelease` equals `branch != default`; the release-create step is gated `exists == 'false' || github.event_name == 'workflow_dispatch'` (the step output is the string `'false'`, not a boolean); the asset-delete step carries that same condition, narrowed by `inputs.expect_release_assets`. Where `workflow_dispatch` is the publisher's only trigger (`releaseTrigger: dispatch-only`) every run is a dispatch, so the exists-check's skip leg can never fire: record that leg N/A rather than failed, and expect the release-create step to still carry the `exists == 'false' || github.event_name == 'workflow_dispatch'` condition, since D6.4 keeps the `github-release` job body verbatim. A caller with no file target passes `expect_release_assets: false`, which covers a Docker-only, a PyPI-only, and a source-only repo alike, while a NuGet-only caller keeps the default `true` because its leaf uploads a `release-asset-*` carrying the package, and a source-only caller also sets every `enable_*` input false. `github-release` and the terminal registry pusher (Docker) each carry `!failure() && !cancelled()` rather than the implicit `success()`, so a failed build skips both, while a target that merely skipped, being disabled or unchanged, still lets the release and the Docker push proceed, and a package target's separate publish job `needs:` the release-task call for the same reason (D4.5). A first `plan` job decides once whether the run publishes, admitting a code-affecting bot push to `main`, a dispatch of `main` or `develop`, and a `main`-only schedule, and every publishing job gates on that decision (D4.1). -- **D5:** each cross-job transfer artifact has a delete step at its consumer, gated so it runs exactly when the consumption happened rather than when the whole job succeeded (D5.2), `continue-on-error: true`, tolerating a failed listing, and looping all ids; **every** upload sets `retention-days: 1`; **no** cleanup step enumerates and deletes the run's whole artifact set, in whatever jq or API shape it is written. -- **D6:** the release download uses `pattern:`/`merge-multiple:` (no `artifact-ids:`). Branch-derived config reads `inputs.branch` (a `github.ref_name` in such config is a finding). Artifact names are branch-suffixed. The target set is consistent across every surface D6.4 names: the `enable_` input, the `build-` job, that job's entries in the `github-release` and `build-docker` `needs:` lists, the `changes` paths-filter entry **and** its output, the `smoke-build` enable-forward, and any separate `publish-` job the package-registry seam requires. The `inputs.branch` rule above binds a called leaf, while a `publish-` job is in the publisher and reads `github.ref_name` correctly. -- **D7:** the publisher concurrency group is global and ref-independent with `cancel-in-progress: false`. A reusable job declares `permissions:` only where every caller grants that scope at startup, per D7.2. A boolean used by both `workflow_call` and `workflow_dispatch` is declared in both trigger blocks, and a boolean read through `github.event.inputs.` is compared against `'true'` alone, a comparison against the boolean `true` never firing on a string, while an `inputs.` read carries the declared boolean and is used directly, a both-forms comparison there being redundant rather than a finding, so a repo whose booleans all arrive by `workflow_call` records the comparison half N/A. A job or step output is a string too and takes the same `== 'true'` (D7.3). Every cross-job condition that admits a skipped dependency pairs its allowlist with a status-check function such as `always()` or `!failure() && !cancelled()`, since the implicit `success()` is false the moment a `needs:` job skipped (D7.4). -- **D8/D9:** the merge-bot enables auto-merge on `opened`/`reopened` for every Dependabot tier including semver-major, dispatches `--squash`/`--merge` by the PR's base ref, disables on a maintainer-pushed `synchronize`, and keys concurrency on the PR number rather than `github.ref`. Codegen runs as a matrix over both branches, and Dependabot targets both branches with security PRs to the default branch (D8.2). The upstream tracker's `bump-branch-prefix` and `branches` match a merge-bot rule, unless it sets `auto-merge: false`, which prefixes the head so no rule matches (wrapper repos). A gate comparing `github.actor` against hard-coded bot identities emits a `::warning::` on the non-matching branch, and the annotation is optional only where the failure announces itself anyway (D8.4). Actions are SHA-pinned. Names, shells, and conditionals follow `GOVERNANCE.md` "Workflow YAML Conventions", and line endings follow `.editorconfig` (D9.5), per the policy `GOVERNANCE.md` "Documentation Style Conventions" routes to under "Line Endings". - -**Per-type addenda (apply only the ones present):** - -- **.NET publish:** the smoke runtime set is a strict non-empty subset of the full runtime set. The selected set runs sequentially inside one composite-action job. A non-smoke run uploads one `release-asset--dotnet-publish` artifact, while a smoke run skips the archive and upload steps. -- **NuGet:** `publish-nuget` is a job in the repo's own publisher, never inside the release task and never in a reusable workflow a different repository hosts. `id-token: write` appears on that job only, absent from the build and PR paths, beside `actions: write` for the artifact cleanup. The push uses `--skip-duplicate` and is gated by that job's `needs:` on the release-task call, never on an existence check, so a PR never reaches it. The job consume-then-deletes `nuget-build-` under the download step's own success, per D5.1 and D5.2. `*.nupkg` push also carries the paired `.snupkg` to the symbol server where symbols are enabled. The `release-asset` `.7z` carries the package(s). -- **PyPI:** `publish-pypi` declares `environment: { name: pypi }`. `id-token: write` appears only on that job (absent from the build/PR path). `skip-existing: true` is set on the publish action. The build artifact is deleted under the download step's own success, per D5.2. The `pypi` environment has a deployment-branch rule. -- **Docker:** the leaf reads the external state file for the tag instead of `SemVer2` (wrapper repos only, since a plain Docker repo correctly tags off `SemVer2` and records this N/A). The readme job is gated main-only, both by the caller's branch input and inside the hub-hosted `publish-docker-readme-task.yml` itself. The docker-readme task validates `repositories` XOR `manifest`+`manifest-jq`. The buildcache follows D9.4. -- **Static site deployed to a host:** the generator is pinned by version **and** by a checksum verified before install, declared once across the workflows that install it. The deploy is a dispatch carrying an environment choice, with concurrency keyed on the **environment** and `cancel-in-progress: false`, and production gated to the default branch while any ref may reach a non-production environment. The reusable callee re-asserts the environment name in a job of its own. The upload targets a per-release directory and carries no delete flag at the environment root, and the pointer flip is a separate step. The terminal check asserts the environment, then the release id, then the URL contract, waiting for convergence to a bounded timeout rather than sampling once, and reporting an unreachable host distinctly from an HTTP status (D4.6). Retention is bounded by a declared count and one side is recorded as owning the prune: the deploy asserts it where the credential can observe the destination, and the host owns it where the credential is confined write-only (D5.6). +Cite what each verdict rests on. That is `file:line` for a file in the audited repo, its own name where a setting, a ruleset, or a secret name rather than a file is the evidence, and `/@` plus the `file:line` in that repo where the guarantee binds a workflow or composite action the audited repo only reaches, read at the SHA the caller pins. An **N/A** verdict names the absent construct instead, there being no line to cite. ### 5B. End-to-End Trace Scenarios (No Execution, Deterministic from the YAML) @@ -245,23 +229,23 @@ For each *applicable* scenario, evaluate every job's `if:`/`needs:` against the | # | Input | Expected output | Exercises | | --- | --- | --- | --- | -| S1 | PR touching a build target | `changes` flags it; validation runs; that target's smoke build runs; no push, **no uploads**; validate-release **skipped (smoke), succeeds**; release **skipped**; aggregator **success**; version = prerelease; no release; no dangling artifacts | D1, D2.2, D3 | +| S1 | PR touching a build target | `changes` flags it; validation runs; that target's smoke build runs; no push, **no uploads**; validate-release **succeeds**, its check exiting early on smoke per D2.2; release **skipped**; aggregator **success**; version = prerelease; no release; no dangling artifacts | D1, D2.2, D3 | | S2 | PR changing only docs | smoke-build **skipped**, validation runs, aggregator **success** | D1.1, D1.2, D1.5 | | S3 | PR changing only `.github/workflows/**` | the filter marks no target -> smoke-build **skipped**, validation runs, aggregator **success** | D1.2, D1.4, D1.5 | -| S4 | PR base = default branch, carrying a build target | smoke versions as prerelease, validate-release **skipped (smoke), succeeds** so the default-branch arm does **not** fire, aggregator **success**, promotion not blocked | D1.5, D2.2, D3.2 | +| S4 | PR base = default branch, carrying a build target | smoke versions as prerelease, validate-release **succeeds** with its check exited early per D2.2, so the default-branch arm does **not** fire, aggregator **success**, promotion not blocked | D1.5, D2.2, D3.2 | | S5 | bot push to `main` not touching a release path (e.g. an Actions bump) | the paths filter excludes it, so nothing publishes | D4.1 | | S6 | code-affecting **bot** push to `main` (a human push/promotion, or any develop push, does not) | the `plan` job gates it to the App/Dependabot actor, and `main` publishes a release | D3, D4 | | S7 | publish run (schedule, a bot push to main, or a dispatch) | builds the **one** trigger branch: `main` -> `X.Y.Z`, `prerelease=false`, registry stable, readme run; `develop` -> `X.Y.Z-g`, `prerelease=true`, registry prerelease; `release-asset-*` consumed-then-deleted; each package build-artifact (`nuget-build-*`, `pypi-build-*`) deleted after its publish; **no dangling artifacts** | D3, D4, D5, D6, D7 | | S8 | dispatch from a ref other than `main` or `develop` | **fails fast** | D2.3 | | S9 | re-run publish on a schedule or push trigger, version unchanged (a dispatch re-run refreshes the release instead, per D4.4) | release-create **skipped**, `release-asset-*` delete **skipped**; NuGet/PyPI pushes no-op (server dedupe); **package build-artifacts still deleted** (their download succeeded); **Docker still re-pushes** the image; no duplicate release | D4.4, D5.2 | | S10 | branch/version classification disagree | validate-release **fails loud**, build/publish skip | D2.2 | -| S11 | scheduled upstream-version bump (wrapper) | resolver detects a change -> commits the state file -> opens a `-` PR -> merge-bot auto-merges -> the `main` pin publishes via the gate (a develop pin does not auto-publish, shipping instead via a develop dispatch or promotion) | D8.3, D3.5 | +| S11 | scheduled upstream-version bump (wrapper) | resolver detects a change -> commits the state file -> opens a per-branch bump PR -> the merge-bot auto-merges it, or leaves it for the maintainer where the tracker sets `auto-merge: false` (D8.3) -> the `main` pin publishes via the gate (a develop pin does not auto-publish, shipping instead via a develop dispatch or promotion) | D8.3, D3.5 | | S12 | deploy dispatch naming an environment | the ref gate runs **first** (production from the default branch only, any ref to a non-production environment); validation runs; the callee re-asserts the environment name; a release installs under its own id; the pointer flips as a separate step; retention is bounded by whichever of the two D5.6 shapes the repo uses, so a deploy whose credential can observe the destination asserts the count converged and one confined write-only leaves it to the host; the live check asserts the environment and the release id, waiting out the reload, then the URL contract; **no tag and no release are created** | D2.1, D4.6, D5.6 | | S13 | deploy dispatch of a production environment from a non-default ref | **fails fast**, before anything is installed or written | D2.1 | ### 5C. Live Probe (Where Warranted) -Every probe here that dispatches a workflow or re-runs a real publish is the maintainer's to run, and the agent prepares the command and reads the result back afterwards. A harness that refuses such a write is the harness working as intended, and the refusal is neither re-shaped into a raw API call nor talked around (GOVERNANCE.md "Repository Boundaries and Write Safety"). +Every probe here that opens a pull request, dispatches a workflow, or re-runs a real publish is the maintainer's to run, with the agent preparing the command and reading the result back afterwards. A harness that refuses such a write is the harness working as intended, and the refusal is neither re-shaped into a raw API call nor talked around (`GOVERNANCE.md` "Repository Boundaries and Write Safety"). - Open a trivial-change PR touching one target and confirm S1. *Caveat: the Docker leg logs in to the registry even on smoke and reads the buildcache, so it needs `DOCKER_HUB_*` secrets and cannot run on a fork PR (same-repo only).* - Per registry: after a real publish, query NuGet.org for the expected version + prerelease classification (and the `.snupkg` on the symbol server), and confirm a re-run added no duplicate. For PyPI read the built `dist/*` filenames out of the build job's log, `.dev0` off `develop` vs a plain version on the default branch. @@ -272,23 +256,23 @@ Every probe here that dispatches a workflow or re-runs a real publish is the mai Record the workflow **operational** when every *applicable* 5A item passes, every *applicable* 5B scenario's predicted output equals the expected, and no 5C probe that was run contradicts either. N/A items are excluded, never counted as failures. Any *applicable* mismatch is a **defect** -> **not operational**. Procedure: -1. **Audit** with 5A. Record pass/fail/N-A with `file:line`. +1. **Audit** with 5A, recording each item's verdict and its evidence in the form 5A sets out. 2. **Trace** the applicable S-scenarios with 5B. Diff predicted vs expected. -3. **Probe** with 5C where a live signal exists that the static trace cannot produce: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. +3. **Probe** with 5C where a live signal exists that the static trace cannot produce, running the probes that only read and preparing the writing ones for the maintainer: live version classification, registry state, the artifact lifecycle of a real run, and the deploy ref gate. 4. **Verdict:** operational / not operational, with the failing guarantee(s) and the triggering input for each, the list of items recorded N/A, and the 5C probes prepared but not run. `WORKFLOW.md` section 5 keeps the test methodology, and the `workflow-ci-contract` Skill at `.agents/skills/workflow-ci-contract/references/test-methodology.md` in the hub, not a repo-relative link since that path is hub-local and not carried into every fleet repo, carries this section whole as a generated include. ## 6. Per-Project-Type Test Walkthroughs -Each type maps the *applicable* S-scenarios onto its targets. The differences are which leaf tasks exist and what each produces, which 5A addenda apply, and which scenarios are N/A. Walking these is the self-check that the contract holds for each shape. +Each type maps the *applicable* S-scenarios onto its targets. The differences are which leaf tasks exist and what each produces, and which guarantees and scenarios are N/A. Walking these is the self-check that the contract holds for each shape. - **.NET publish.** The target runs a sequential `dotnet publish` runtime loop inside one composite-action job. Configuration is Release on the default branch and Debug otherwise. A non-smoke run builds the full runtime set, archives the combined output as a `.7z`, and uploads it as `release-asset--dotnet-publish`. The archive is named from the project file stem unless `dotnet_publish_asset_name` overrides it. A smoke run builds a two-runtime subset and skips the archive and upload steps, so it uploads nothing. S1 smoke-builds that subset after a .NET project change. S7 attaches the 7z from a non-smoke run. The non-default leg sets `prerelease=true`, and the default leg sets `prerelease=false`. GitHub marks the stable default release "Latest" automatically. - **NuGet.** The leaf uploads both `release-asset--nuget` and `nuget-build-` on a non-smoke run and pushes nothing, and a separate `publish-nuget` job in the repo's own publisher consumes the second and runs `dotnet nuget push *.nupkg --skip-duplicate`, then deletes it under the download step's own success (D5.2). Section 3's package-registry seam says why the push sits there rather than in the leaf. Configuration is Release on the default branch, Debug otherwise. Where symbols are enabled (`snupkg`), the push auto-carries the paired `.snupkg` to NuGet.org's symbol server and the release-asset `.7z` also contains it, a triple surface. NuGet.org derives `isPrerelease` from the SemVer2 `-g` suffix (the workflow sets no such flag). Test: S7 non-default leg publishes a prerelease package + asset, default a stable; S9 re-run is a server-side `--skip-duplicate` no-op. 5C: query NuGet.org for both versions and the symbol package. - **PyPI.** The leaf builds and uploads `pypi-build-`. A **separate** `publish-pypi` job (with `environment: pypi`, `id-token: write`, `actions: write`) does the OIDC Trusted-Publishing upload with `skip-existing: true`, then **consume-then-deletes** the build artifact under the download step's own success (D5.2), so on S9 it is deleted even though the `release-asset-*` delete is skipped. The version is `AssemblyFileVersion` with `.dev0` appended on `develop` only, and must stay `--pre`-selectable and sorted above the default release. PyPI contributes no `release-asset-*`. A PyPI-only repo sets `expect_release_assets: false` at the caller. Test: S7 default leg publishes a release, non-default a `.dev0`; S9 is a `skip-existing` no-op; 5C inspects the `dist/*` filenames and the compute-version log. - **Docker image.** The leaf pushes the default branch multi-arch (amd64+arm64) and any other branch `amd64`-only, with a per-branch registry buildcache (`buildcache-`; a multi-image repo adds a per-image tag) (`cache-to` only the built branch and only on push, `cache-from` both branches); no `release-asset-*`, so a Docker-only repo's caller passes `expect_release_assets: false`; the readme job (`peter-evans/dockerhub-description`, `DOCKER_HUB_ACCESS_TOKEN`) runs **only** when the default branch publishes, whether called directly or reached through the hub-hosted `publish-docker-readme-task.yml`; the docker-readme task validates `repositories` XOR `manifest`+`manifest-jq` and a multi-image repo derives its publish matrix from the manifest. Docker **always re-pushes** the image, independently of a skipped release-create (S9). A **wrapper** repo tracks an upstream release: the upstream tracker writes a `name -> version` state file and the merge-bot auto-merges the bump PR (S11), and the leaf MUST read that file for the immutable tag instead of `SemVer2` (the tracker ships without this consumer wiring). Test: S7 default leg pushes `latest` + the version tag and updates the readme. Non-default pushes the develop tag (amd64 only). S9 still re-pushes. S11 ships the bumped upstream version next publish. 5C Docker probe needs `DOCKER_HUB_*` secrets and same-repo (not fork) runs. -- **Data / asset library.** A single new leaf: validate -> zip -> upload `release-asset--library` (`retention-days: 1` per D5.4, upload gated on smoke being false per D1.3). Because no such leaf ships, you **add a target** (D6.4): a new `enable_library` input + `build-library` job + its `github-release` and `build-docker` `needs:` entries in the release task, and a `library` paths-filter entry + `changes` output + `smoke-build` enable-forward in the PR workflow (without it, D1.1 never smoke-builds the library). Keep `expect_release_assets: true` (it has a file target, unlike Docker). The caller's own validation job is replaced by a type-appropriate validator only where the reusable one cannot express this repo's validation, with the aggregator re-pointed to the replacement (D1.2). `smoke-build` keeps `needs: [changes]`, as D1.2 and the hub's release-with-smoke stub both have it. `version.json` + the NBGV `get-version` step are retained (they own the tag). Test: S1 smoke runs validate+zip and uploads nothing; S7 attaches the zip, prerelease on the non-default leg; S9 on a *scheduled* re-run release-create + asset-delete skip (the existing zip is untouched, no registry push), while a `workflow_dispatch` re-run **refreshes** the release and re-runs the asset-delete (the asset is re-uploaded then re-deleted). N/A: the NuGet, PyPI, Docker, and .NET publish 5A addenda and their scenario clauses. -- **Source-only / no build.** There is no package/image build leaf. A repo may own the reusable release task or call its hub-hosted copy. The dispatch-only `publish-release.yml` reaches the reusable plan, validation, and release tasks. Its publish job passes `github: true`, every `enable_*` input as false, and `expect_release_assets: false`. This produces tag + source zip + README + LICENSE with no asset download. With no target, the paths-filter matches nothing. A retained `smoke-build` job is therefore **structurally always skipped**. The repo may instead drop that never-running job. Validation remains the caller's own job reaching the reusable validator. The aggregator `needs:` that validation job (D1.2), and a retained `smoke-build` job `needs:` the `changes` job rather than the validation job. NBGV and `version.json` own the tag. The publish job depends on the same reusable validation task that the PR workflow runs. This prevents a dispatch from releasing a ref that fails validation. Applicable scenarios are S1 (validation only), S7, S8, and S10. S7 covers the release, S8 the dispatch guard, and S10 the classification gate. S9 is recorded N/A, since S9's input is a schedule or push re-run and this publisher triggers on neither, so its no-op skip leg can never fire. S2-S6, D5/D6 artifact items, and all per-type 5A addenda are N/A. The artifact-lifecycle and registry clauses of S7 are also N/A, not failed. +- **Data / asset library.** A single new leaf: validate -> zip -> upload `release-asset--library` (`retention-days: 1` per D5.4, upload gated on smoke being false per D1.3). Because no such leaf ships, you **add a target** (D6.4): a new `enable_library` input + `build-library` job + its `github-release` and `build-docker` `needs:` entries in the release task, and a `library` paths-filter entry + `changes` output + `smoke-build` enable-forward in the PR workflow (without it, D1.1 never smoke-builds the library). Keep `expect_release_assets: true` (it has a file target, unlike Docker). The caller's own validation job is replaced by a type-appropriate validator only where the reusable one cannot express this repo's validation, with the aggregator re-pointed to the replacement (D1.2). `smoke-build` keeps `needs: [changes]`, as D1.2 and the hub's release-with-smoke stub both have it. `version.json` + the NBGV `get-version` step are retained (they own the tag). Test: S1 smoke runs validate+zip and uploads nothing; S7 attaches the zip, prerelease on the non-default leg; S9 on a *scheduled* re-run release-create + asset-delete skip (the existing zip is untouched, no registry push), while a `workflow_dispatch` re-run **refreshes** the release and re-runs the asset-delete (the asset is re-uploaded then re-deleted). N/A: the guarantees of the NuGet, PyPI, Docker, and .NET publish targets, and their scenario clauses. +- **Source-only / no build.** There is no package/image build leaf. A repo may own the reusable release task or call its hub-hosted copy. The dispatch-only `publish-release.yml` reaches the reusable plan, validation, and release tasks. Its publish job passes `github: true`, every `enable_*` input as false, and `expect_release_assets: false`. This produces tag + source zip + README + LICENSE with no asset download. With no target, the paths-filter matches nothing. A retained `smoke-build` job is therefore **structurally always skipped**. The repo may instead drop that never-running job. Validation remains the caller's own job reaching the reusable validator. The aggregator `needs:` that validation job (D1.2), and a retained `smoke-build` job `needs:` the `changes` job rather than the validation job. NBGV and `version.json` own the tag. The publish job depends on the same reusable validation task that the PR workflow runs. This prevents a dispatch from releasing a ref that fails validation. Applicable scenarios are S1 (validation only), S7, S8, and S10. S7 covers the release, S8 the dispatch guard, and S10 the classification gate. S9 is recorded N/A, since S9's input is a schedule or push re-run and this publisher triggers on neither, so its no-op skip leg can never fire. S2-S6, D5.1-D5.5, D6.1 and D6.3, and the guarantees of the .NET publish, NuGet, PyPI, Docker, and data-library targets are N/A. The artifact-lifecycle and registry clauses of S7 are also N/A, not failed. - **Static site deployed to a host the project owns.** Two independent surfaces, and keeping them apart is the point. The **release** is the source-only shape above, unchanged: a dispatch-only `publish-release.yml` where NBGV and `version.json` own the tag, producing tag + source zip + README + LICENSE. The **deploy** is its own `workflow_dispatch` carrying an `environment` choice input, so redeploying an unchanged commit mints no tag, which matters because redeploying is routine. It runs a ref gate **first**, before anything is installed or written (production from the default branch only, while any ref may reach a non-production environment, since proving a branch before it merges is what that environment is for), then the **same** reusable validation task the PR gate runs, so a dispatch cannot deploy a ref that fails validation, then calls the hub-hosted `deploy-site-task.yml`, with the `environment:` declared inside that task rather than on the calling job, since GitHub rejects a job carrying both `uses:` and `environment:`. The crossing secrets, `DEPLOY_SSH_PRIVATE_KEY` and the optional `SITE_AUTH_TOKEN_ID`/`SITE_AUTH_TOKEN` pair the live check needs, are therefore mapped explicitly under the call's `secrets:`, the pair only where a token-gated live check needs it, since the task declares them and `secrets: inherit` is not used on a cross-repository call. What that task's own job reads for each of them comes from its `environment:` binding rather than from the caller's job context. Concurrency is keyed on the environment with `cancel-in-progress: false`, because a cancelled deploy leaves a release uploaded and unflipped. The task re-asserts the environment name in a job of its own, because the `environment:` binding resolves before any step runs and a `workflow_call` caller is not bound by the dispatch choice list a human sees. Its environment-bound job then: checks out full history (a shallow clone silently changes page metadata), derives the release id **once** and exports it (deriving it twice yields ids seconds apart, and the live check then asserts a version nothing installed), runs a required deploy hook that builds the tree with whatever generator and precompression the site owns, installs the deploy credential from the environment, uploads into a per-release directory hard-linked against the current release and carrying **no** delete flag (at an environment root a delete removes the rollback targets), flips the pointer as a separate atomic step so a failed transfer cannot half-publish, then runs the same hook again to prune old releases and to check the running host (D4.6). Retention (D5.6) is bounded by a declared count with one side recorded as owning it: a deploy whose credential can observe the destination prunes and asserts the count here, while a credential confined **write-only** can neither delete nor read back, so there the prune is a host-side timer and the repo's runbook records that ownership. Widening the credential to bring the prune in-pipeline would trade a real confinement boundary for a check, and is the wrong trade. What the guarantee rejects is neither side owning it. One thing the pipeline cannot assert and the server config must: a non-public environment serving a byte-identical copy must not be indexed, and that default belongs on the side that is harmless in production, since a non-public container missing the value is still behind its gate while a production container inheriting it deindexes the site silently. Applicable scenarios: S1 (validation), the source-only release set S7/S8/S10, and S12/S13 (the deploy dispatch). N/A: S2-S4, S9, every registry scenario, and D5.1-D5.4 (the pipeline uploads no workflow artifact at all, so D5.6 is what applies in their place), all recorded N/A, not failed. - **Operational (workflow model, not a build target).** A `workflowModel: operational` repo layers direct commits to `develop` onto the **source-only** release shape above. It has two workflows. The first is a **lint/validation** PR workflow that feeds the required `Check pull request workflow status job`. It uses the generic linters (editorconfig/EOL, markdownlint, cspell, actionlint) plus a domain validator, with **no unit tests**. Examples include Home Assistant `hass --script check_config`, `esphome config`, or a firmware build. Its triggers differ from the `release` model. It runs on pushes to `develop`, pull requests to `[ main, develop ]`, and `workflow_dispatch`. Push validation is advisory. Pull request validation is enforced on `main` and reported but not required on `develop`. The second workflow is the standard **source-only publisher** with `releaseTrigger: dispatch-only`. NBGV and `version.json` own the tag. The reusable release task creates tag + source zip + README + LICENSE. **The PR trigger names both branches, and naming `main` alone is a defect.** Omitting `develop` starts no validation when a PR opens against `develop`. The aggregator then never reports, and the PR appears clean with an empty check list. D1.2 forbids that output. Naming both causes a duplicate run after a PR merge. The change validates on the PR and again on the resulting push, regardless of merge method. The operational `develop` ruleset prescribes no merge method. The concurrency group uses the workflow name plus `${{ github.ref }}` (`GOVERNANCE.md` "Workflow YAML Conventions"). A pull request uses `refs/pull//merge`, while its push uses `refs/heads/develop`. The runs occupy different groups and neither cancels the other. Pay that cost. The lint-only gate costs only a few runner-minutes. Suppressing the push requires distinguishing a merge commit from a direct commit, which restores the ambiguity the trigger set removes. S1 applies to every PR, including promotion and `develop` PRs. The source-only S7, S8, and S10 scenarios also apply, with S9 recorded N/A for the same dispatch-only reason. Bot-push and schedule paths in S5/S6 are N/A, as are every build and registry scenario. See the branch-model note in Section 3 and [GOVERNANCE.md "Branching Model"][governance-branching-model]. diff --git a/reports/canonical-review.json b/reports/canonical-review.json index 27cd6a24..b13fede1 100644 --- a/reports/canonical-review.json +++ b/reports/canonical-review.json @@ -49,6 +49,14 @@ "hubCommit": "6aecb8b9b68b04156701a67ee14fbf5144b86332", "stamp": "2026-09-05T14:16:35Z" }, + { + "unit": ".agents/skills/audit-a-repo/SKILL.md > Measuring", + "digest": "sha256:641112a85d02d327be4e7d1154b26a19f116a8087c993de516a760857b1a6ed4", + "reviewer": "agent-skill", + "findings": 0, + "hubCommit": "2d63d52cbb8bd8a6d42846ddbdda18737c069187", + "stamp": "2026-09-06T20:51:13Z" + }, { "unit": ".agents/skills/backlog-burndown/SKILL.md > (preamble)", "digest": "sha256:75ac6fdd150652041b8cf1c002d40d7e64903e394983f6ae0c62bec07e200d42", @@ -515,11 +523,11 @@ }, { "unit": ".agents/skills/workflow-ci-contract/SKILL.md > How the Contract Is Read", - "digest": "sha256:b208f86efb05ab8e3ff139f25e4610487358fa03df0222a224c540cd14882122", + "digest": "sha256:d1c4af2c63eae9841c3ca5755a76f94f290367f5fb7cb5be1810d7e3be5ef028", "reviewer": "agent-skill", "findings": 0, - "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", - "stamp": "2026-09-06T18:19:28Z" + "hubCommit": "2d63d52cbb8bd8a6d42846ddbdda18737c069187", + "stamp": "2026-09-06T20:18:39Z" }, { "unit": ".agents/skills/workflow-ci-contract/SKILL.md > Style Rules", @@ -587,11 +595,11 @@ }, { "unit": ".agents/skills/workflow-ci-contract/references/test-methodology.md > (preamble)", - "digest": "sha256:9429013b2fc946cf54d1eb40ca5e3756211ca3e7992911ac50fb837e325e3771", + "digest": "sha256:12773b882e5a37f82e1f710eb8081887a46daf54e7da5535509f46a77d24d18d", "reviewer": "agent-skill", - "findings": 2, - "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", - "stamp": "2026-09-06T17:51:06Z" + "findings": 0, + "hubCommit": "2d63d52cbb8bd8a6d42846ddbdda18737c069187", + "stamp": "2026-09-06T20:21:42Z" }, { "unit": ".agents/skills/workflow-ci-contract/references/test-methodology.md > 5A: Static Audit", @@ -619,11 +627,11 @@ }, { "unit": ".agents/skills/workflow-ci-contract/references/test-methodology.md > The Test Methodology", - "digest": "sha256:a084b10818b512d3c6696aae3d678955cbe27290631046e173d2cdd035b59cc1", + "digest": "sha256:5194ff2a1f7bb85ef300bddf83016e3d96807d2a5fed9533414cf928109f8037", "reviewer": "agent-skill", "findings": 0, - "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", - "stamp": "2026-09-06T17:51:07Z" + "hubCommit": "2d63d52cbb8bd8a6d42846ddbdda18737c069187", + "stamp": "2026-09-06T20:18:39Z" }, { "unit": ".agents/skills/workflow-ci-contract/references/test-methodology.md > Verdict", @@ -657,6 +665,14 @@ "hubCommit": "66d79460335379301db83fe6870303f8f701d92b", "stamp": "2026-09-06T05:06:22Z" }, + { + "unit": "AUDIT.md > (preamble)", + "digest": "sha256:a2ef5c014adb4303ae7eac8a6bdf8569964529a5733ef6ce65c35efef6147454", + "reviewer": "agent-skill", + "findings": 1, + "hubCommit": "2d63d52cbb8bd8a6d42846ddbdda18737c069187", + "stamp": "2026-09-06T20:42:51Z" + }, { "unit": "AUDIT.md > 10. Converge: Apply the Fixes", "digest": "sha256:6bc1e573e6c9cd6dace6df67a0d826aaa910d8d94cdf0f29d7571c53def4ff34", @@ -665,6 +681,14 @@ "hubCommit": "c3b2898feef97459d01ce7c0b63de25b6c5524bf", "stamp": "2026-09-01T13:57:20Z" }, + { + "unit": "AUDIT.md > 3. Applicability Gate", + "digest": "sha256:5a93264a923298617dc9ed670ff4a3acd3640012370d8453904825083ac2c0bd", + "reviewer": "agent-skill", + "findings": 0, + "hubCommit": "2d63d52cbb8bd8a6d42846ddbdda18737c069187", + "stamp": "2026-09-06T20:42:51Z" + }, { "unit": "AUDIT.md > 4. Per-Dimension Checks (Letter and Intent)", "digest": "sha256:083d3ca4108c8b1d047e086924360dba45106e62aad5546636a6cbf468271143", @@ -673,6 +697,14 @@ "hubCommit": "78898becaa2b1a62cb4c806d86273210c6390ac5", "stamp": "2026-09-01T14:48:23Z" }, + { + "unit": "AUDIT.md > 5. Assert the Actions Implement WORKFLOW.md", + "digest": "sha256:bc28245e825a254487945189714d112ef5e5410e5967dcfd4ee79c2d3ce398f5", + "reviewer": "agent-skill", + "findings": 1, + "hubCommit": "2d63d52cbb8bd8a6d42846ddbdda18737c069187", + "stamp": "2026-09-06T19:30:43Z" + }, { "unit": "AUDIT.md > 6. Validate Settings, Rulesets, and Secrets", "digest": "sha256:cadbf5a3b36bdc9ec871eb907565c582bed70c2e4098c6a99fd756cd2b623489", @@ -755,11 +787,11 @@ }, { "unit": "WORKFLOW.md > 1. Purpose and How to Use This Document", - "digest": "sha256:35a8c77043da983af04a46300f652d4e3b8a454815b10c380524832cfdbe4a70", + "digest": "sha256:326b1daa6187f5675ddeefbd5d8bd6fb1510e861d158f152c794a88c689824c2", "reviewer": "agent-skill", "findings": 0, - "hubCommit": "c3b2898feef97459d01ce7c0b63de25b6c5524bf", - "stamp": "2026-09-01T13:57:20Z" + "hubCommit": "2d63d52cbb8bd8a6d42846ddbdda18737c069187", + "stamp": "2026-09-06T19:39:11Z" }, { "unit": "WORKFLOW.md > 2. Workflow Style Conventions", @@ -787,19 +819,19 @@ }, { "unit": "WORKFLOW.md > 5. Test Methodology", - "digest": "sha256:0bdfdb5d7fc20469237f06286a59d36d245192c15310ae55b245196f13206eec", + "digest": "sha256:03e87dcc7b8a745aab8dccc8b2bd7558e948f880e31f1c063074dae9b2ebfd6f", "reviewer": "agent-skill", - "findings": 6, - "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", - "stamp": "2026-09-06T17:51:07Z" + "findings": 1, + "hubCommit": "2d63d52cbb8bd8a6d42846ddbdda18737c069187", + "stamp": "2026-09-06T20:18:39Z" }, { "unit": "WORKFLOW.md > 6. Per-Project-Type Test Walkthroughs", - "digest": "sha256:ade51d875b83dfe761ec121d307a06c21e9ba730eaa2ff27e4e428833ee8cfdf", + "digest": "sha256:740b093d556b31d42f5edb30c09c718c5a2f38d3c9eb87c7287743ae2537cb15", "reviewer": "agent-skill", - "findings": 5, - "hubCommit": "0f9114282e9202db6f22a1ab23a9e8b73f02be11", - "stamp": "2026-09-06T17:51:07Z" + "findings": 4, + "hubCommit": "2d63d52cbb8bd8a6d42846ddbdda18737c069187", + "stamp": "2026-09-06T21:02:39Z" } ] }