Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/publish-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ jobs:
publish:
name: Publish project release job
needs: [ plan, validate ]
if: ${{ needs.plan.outputs.publish == 'true' }}
if: ${{ needs.plan.outputs.publish == 'true' && needs.validate.result == 'success' }}
uses: ./.github/workflows/build-release-task.yml
permissions:
contents: write
Expand Down
14 changes: 9 additions & 5 deletions AUDIT.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,15 +95,15 @@ Run [`WORKFLOW.md`][workflow]'s methodology against the repo's **own** Actions:

- **General settings** - diff the live repository settings against [`repo-config/settings.json`][repo-config-settings], and confirm the two state-dependent settings: `has_discussions` follows visibility (public on / private off) and `default_branch` is `main`.

```sh
```bash
live=$(gh api "repos/<owner>/<repo>" --jq '{has_wiki,has_projects,allow_merge_commit,allow_squash_merge,allow_rebase_merge,allow_auto_merge,allow_update_branch,delete_branch_on_merge}')
diff <(jq -S . repo-config/settings.json) <(jq -S . <<<"$live") \
&& echo "settings: in sync" || echo "settings: DRIFT"
```

- **Rulesets** - diff each live ruleset against the committed expected payload with a normalized comparison (sort the order-insensitive `rules[]` on each rule's whole content before diffing, so a reordered but equivalent ruleset does not read as drift). The compared subset is `name`, `target`, `enforcement`, `conditions` and `rules`, and `bypass_actors` sits deliberately outside it, which is the same subset and the same sort key [`spec/audit.py`][audit-runner] uses. Who may bypass a ruleset is a per-repository human decision taken in the UI, no payload declares one, and [`repo-config/configure.sh`][repo-config] treats it that way in both modes, writing the live list back unchanged on `apply` and reporting it without asserting on `check`. Comparing it here would contradict that and report a ruleset finding against every repository that has any bypass actor, which is the field's normal state rather than a deviation:

```sh
```bash
# bypass_actors stays outside the projection, since no payload declares one and jq cannot sort the null that leaves.
# Rules sort on each rule's whole content, matching the key normalize_ruleset in audit.py sorts by.
# Sorting on .type alone leaves two rules of one type in input order, so a reordered pair would read as drift.
Expand Down Expand Up @@ -135,10 +135,14 @@ Run [`WORKFLOW.md`][workflow]'s methodology against the repo's **own** Actions:

- **Dependabot ecosystem coverage** - for each ecosystem the repo's tree implies, `.github/dependabot.yml` must declare it: `github-actions` when `.github/workflows/` is present (its workflows reference actions, and otherwise those versions go stale and a stood-up merge-bot has no action-update PRs to auto-merge), and `devcontainers` when a `.devcontainer` is present. The mechanical check (`spec/audit.py`) asserts each implied ecosystem's **presence**. A tree-implied ecosystem declared nowhere is a **drift finding** (the file exists, so its absence would instead be a file-presence letter). Then confirm **by inspection** that each declared ecosystem **dual-targets `main` + `develop`** per the [Branching Model][governance-branching-model], since the regex below cannot pair an ecosystem with its `target-branch`. Language ecosystems (`nuget`/`uv`/`npm`) are directory-scoped and audited by inspection too.

```sh
```bash
# Anchor to the line start (optional list dash) so a commented-out '# package-ecosystem:' is not counted.
decl=$(gh api "repos/<owner>/<repo>/contents/.github/dependabot.yml?ref=<ground>" --jq '.content' | base64 -d | grep -oE '^[[:space:]]*-?[[:space:]]*package-ecosystem:[[:space:]]*"?[a-z-]+' | grep -oE '[a-z-]+$' | sort -u)
has() { gh api "repos/<owner>/<repo>/contents/$1?ref=<ground>" >/dev/null 2>&1; }
dependabot_content=$(gh api "repos/<owner>/<repo>/contents/.github/dependabot.yml?ref=<ground>" --jq '.content') || exit 1
dependabot_yaml=$(base64 -d <<<"$dependabot_content") || exit 1
decl=$(grep -oE '^[[:space:]]*-?[[:space:]]*package-ecosystem:[[:space:]]*"?[a-z-]+' <<<"$dependabot_yaml" | grep -oE '[a-z-]+$' | sort -u)
root_paths=$(gh api "repos/<owner>/<repo>/contents?ref=<ground>" --jq '.[].path') || exit 1
github_paths=$(gh api "repos/<owner>/<repo>/contents/.github?ref=<ground>" --jq '.[].path') || exit 1
has() { grep -Fxq "$1" <<<"$root_paths"$'\n'"$github_paths"; }
has .github/workflows && { grep -qx github-actions <<<"$decl" && echo "github-actions: present" || echo "github-actions: MISSING (workflows present)"; }
has .devcontainer && { grep -qx devcontainers <<<"$decl" && echo "devcontainers: present" || echo "devcontainers: MISSING (.devcontainer present)"; }
# then read dependabot.yml and confirm each present ecosystem has both a main and a develop target-branch entry
Expand Down
115 changes: 115 additions & 0 deletions scripts/tests/test_release_guards.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
#!/usr/bin/env python3
"""Protect release and audit boundaries from fail-open regressions."""

from __future__ import annotations

import unittest
from pathlib import Path
from subprocess import run

REPO = Path(__file__).resolve().parents[2]


class ReleaseGuardCase(unittest.TestCase):
"""Publishing and audit discovery require their prerequisite checks to succeed."""

def test_pypi_artifact_name_matches_contracts_and_consumers(self) -> None:
canonical_name = "pypi-build-"
legacy_name = "pypilibrary" + "-build-"
required_paths = (
"GOVERNANCE.md",
"WORKFLOW.md",
".github/actions/pypi-build-default/action.yml",
"docs/reusable-workflows.md",
)

for relative_path in required_paths:
with self.subTest(path=relative_path):
content = (REPO / relative_path).read_text(encoding="utf-8")
self.assertIn(canonical_name, content)

tracked_text = run(
["git", "grep", "-n", legacy_name],
cwd=REPO,
check=False,
capture_output=True,
text=True,
)
self.assertEqual("", tracked_text.stdout)
self.assertEqual(1, tracked_text.returncode)

def test_publish_requires_successful_validation(self) -> None:
workflow = (REPO / ".github/workflows/publish-release.yml").read_text(encoding="utf-8")
files_spec = (REPO / "spec/files.json").read_text(encoding="utf-8")

self.assertIn(
"if: ${{ needs.plan.outputs.publish == 'true' && needs.validate.result == 'success' }}",
workflow,
)
self.assertIn("\"needs.validate.result == 'success'\"", files_spec)

def test_audit_probes_fail_before_local_path_checks(self) -> None:
audit = (REPO / "AUDIT.md").read_text(encoding="utf-8")
lines = audit.splitlines()
start = next(i for i, line in enumerate(lines) if line.startswith(" dependabot_content="))
probe = "\n".join(line.removeprefix(" ") for line in lines[start : start + 6])
fake_api = r"""
gh() {
case "$2" in
repos/*/contents/.github/dependabot.yml\?*) printf '%s\n' '- package-ecosystem: github-actions' '- package-ecosystem: devcontainers' | base64 ;;
repos/*/contents/.github\?*) printf '%s\n' .github/dependabot.yml .github/workflows ;;
repos/*/contents\?*) printf '%s\n' .devcontainer .github ;;
*) return 17 ;;
esac
}
"""

success = run(
["bash", "-c", f"{fake_api}\n{probe}\nhas .github/workflows && has .devcontainer"],
check=False,
)
failure = run(
["bash", "-c", f"gh() {{ return 17; }}\n{probe}\nexit 0"],
check=False,
)
decode_failure = run(
["bash", "-c", f"gh() {{ printf invalid; }}\n{probe}\nexit 0"],
check=False,
)

self.assertEqual(0, success.returncode)
self.assertNotEqual(0, failure.returncode)
self.assertNotEqual(0, decode_failure.returncode)
self.assertNotIn(
'gh api "repos/<owner>/<repo>/contents/$1?ref=<ground>" >/dev/null 2>&1',
audit,
)

def test_audit_bash_blocks_are_not_labeled_as_posix_shell(self) -> None:
audit_lines = (REPO / "AUDIT.md").read_text(encoding="utf-8").splitlines()
bash_only = ("<(", "<<<", "$'", "[[")
mislabeled = []
fence_label = ""
fence_start = 0
fence_lines: list[str] = []

for number, line in enumerate(audit_lines, start=1):
stripped = line.strip()
if not fence_label and stripped.startswith("```"):
fence_label = stripped.removeprefix("```").split(maxsplit=1)[0]
fence_start = number
elif fence_label and stripped == "```":
if fence_label in {"sh", "shell"} and any(
token in "\n".join(fence_lines) for token in bash_only
):
mislabeled.append((fence_start, fence_label))
fence_label = ""
fence_lines = []
elif fence_label:
fence_lines.append(line)

self.assertEqual([], mislabeled)


if __name__ == "__main__":
unittest.main()
14 changes: 11 additions & 3 deletions spec/audit.py
Original file line number Diff line number Diff line change
Expand Up @@ -2267,7 +2267,7 @@ def _selftest():
" publish:\n"
" name: Publish project release job\n"
" needs: [plan, validate]\n"
" if: ${{ needs.plan.outputs.publish == 'true' }}\n"
" if: ${{ needs.plan.outputs.publish == 'true' && needs.validate.result == 'success' }}\n"
" uses: acme/hub/.github/workflows/build-release-task.yml@" + "a" * 40 + " # 2.0.1\n"
" with:\n"
" github: true\n"
Expand All @@ -2277,7 +2277,7 @@ def _selftest():
"requireTokensInJob": {
"plan": ["publish-plan-task.yml"],
"validate": ["validate-task.yml"],
"publish": ["build-release-task.yml"],
"publish": ["build-release-task.yml", "needs.validate.result == 'success'"],
},
}
cases = [
Expand Down Expand Up @@ -2428,6 +2428,7 @@ def _selftest():
" publish:\n"
" name: Publish project release job\n"
" needs: [validate]\n"
" if: ${{ needs.validate.result == 'success' }}\n"
" uses: acme/hub/.github/workflows/build-release-task.yml@"
+ "a" * 40
+ " # 2.0.1\n",
Expand All @@ -2442,7 +2443,8 @@ def _selftest():
" uses: acme/hub/.github/workflows/publish-plan-task.yml@" + "a" * 40 + " # 2.0.1\n"
" publish:\n"
" name: Publish project release job\n"
" needs: [plan]\n"
" needs: [plan, validate]\n"
" if: ${{ needs.validate.result == 'success' }}\n"
" uses: acme/hub/.github/workflows/build-release-task.yml@"
+ "a" * 40
+ " # 2.0.1\n",
Expand All @@ -2467,6 +2469,12 @@ def _selftest():
publish_contract,
1,
),
(
"publish-release.yml stub whose publish job ignores failed validation",
publish_stub.replace(" && needs.validate.result == 'success'", ""),
publish_contract,
1,
),
]
# The deploy-site.yml caller stub once deploy-site-task.yml is hub-hosted: no secrets: inherit
# (a cross-repository reusable workflow cannot use it), the one crossing secret named instead.
Expand Down
2 changes: 1 addition & 1 deletion spec/files.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
{ "path": "spec/secrets.json", "fidelity": "intent", "intentRef": "docs/repo-config-carry.md", "appliesTo": "*" },
{ "path": ".github/dependabot.yml", "appliesTo": "*" },
{ "path": ".github/workflows/test-pull-request.yml", "fidelity": "interface", "contract": { "requiredJobKeys": ["check-workflow-status", "validate"], "requiredCheckName": "Check pull request workflow status job", "requireTokensInJob": { "validate": ["validate-task.yml"] } }, "intentRef": "GOVERNANCE.md#workflow-yaml-conventions", "appliesTo": "*" },
{ "path": ".github/workflows/publish-release.yml", "fidelity": "interface", "contract": { "requiredJobKeys": ["plan", "validate", "publish"], "requireTokensInJob": { "plan": ["publish-plan-task.yml"], "validate": ["validate-task.yml"], "publish": ["build-release-task.yml"] } }, "intentRef": "WORKFLOW.md#d4---release--publish", "appliesTo": ["two-phase", "dispatch-only", "publish-on-merge"] },
{ "path": ".github/workflows/publish-release.yml", "fidelity": "interface", "contract": { "requiredJobKeys": ["plan", "validate", "publish"], "requireTokensInJob": { "plan": ["publish-plan-task.yml"], "validate": ["validate-task.yml"], "publish": ["build-release-task.yml", "needs.validate.result == 'success'"] } }, "intentRef": "WORKFLOW.md#d4---release--publish", "appliesTo": ["two-phase", "dispatch-only", "publish-on-merge"] },
{ "path": ".github/workflows/merge-bot-pull-request.yml", "fidelity": "interface", "contract": { "requiredJobKeys": ["merge-bot"], "requireTokensInJob": { "merge-bot": ["merge-bot-task.yml", "CODEGEN_APP_CLIENT_ID", "CODEGEN_APP_PRIVATE_KEY"] } }, "intentRef": "WORKFLOW.md#d8---bots--automation", "appliesTo": "*" },
{ "path": ".github/workflows/deploy-site.yml", "fidelity": "interface", "contract": { "requiredJobKeys": ["assert-ref", "validate", "deploy"], "requireTokensInJob": { "deploy": ["deploy-site-task.yml", "\n environment:", "contents: read", "DEPLOY_SSH_PRIVATE_KEY"] } }, "intentRef": "docs/reusable-workflows.md#adopting-the-type-specific-tasks", "appliesTo": ["hugo"] },
{ "path": ".vscode/tasks.json", "sections": ["clean-compile task group"], "reference": "catalog/snippets/configs/vscode-tasks.json", "appliesTo": ["csharp"] },
Expand Down