Skip to content

fix(bin): resolve the CI required-suite roster per repository - #7

Merged
x45dev merged 2 commits into
mainfrom
fm/fm-ci-verify-hardcoded-to-firstmate
Aug 30, 2026
Merged

x45dev merged 2 commits into
mainfrom
fm/fm-ci-verify-hardcoded-to-firstmate

Conversation

@x45dev

@x45dev x45dev commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Intent

Generalize bin/fm-pr-ci-verify.sh so it can confirm a pull request is genuinely green on ANY repository, not only on firstmate itself.

The bug: FM_CI_REQUIRED_SUITES in bin/fm-ci-checks-lib.sh was a hardcoded constant listing firstmate's own twelve CI job names verbatim, including jobs such as 'Behavior tests (Herdr)' and 'Stock macOS Bash snapshot compatibility' that exist nowhere else. Any project whose CI defines its own job names therefore failed the required-suite completeness check by construction, however genuinely green it was. This is not roster drift: on another repository the roster simply does not describe it at all. This was confirmed hitting four consecutive real pull requests (workspace-template, go-bip39-validator, home-fintech twice), each independently hand-verified green and merged on that manual evidence because the shared tool could not confirm them. fm-bearings-snapshot.sh read every non-firstmate PR row as incomplete for the same reason.

The fix: fm_ci_roster now resolves the required roster per repository, and the classifiers take it as an argument rather than reading a constant. Firstmate's own repo must keep working exactly as it does today - this is a generalization, not the removal of a special case.

Deliberate decisions a reviewer reading only the diff would not know:

  • SUBSTITUTION, deliberate and already flagged for review: the original task suggested parsing job names out of the target repository's .github/workflows/ci.yml. That file is the definition but not a roster, because a job name is a template GitHub evaluates (for example 'Behavior portable serial ${{ matrix.shard }}', or a matrix.include leg's '${{ matrix.name }}'), so parsing it would mean reimplementing matrix expansion and the Actions expression language. One of the affected repositories needs exactly that. The roster is therefore derived by OBSERVATION instead: from the job names of the newest successful CI run on the branch the change targets, which GitHub has already expanded exactly. This was a considered tradeoff, not an oversight.
  • No new dependency: gh and jq only, both already required. The accepted cost is up to three GitHub API reads per repository per verification.
  • FM_CI_REQUIRED_SUITES deliberately survives as a documented override, for a change that intentionally adds or removes a CI job.
  • Failing closed is intentional throughout: jq refuses to compile a program whose variables are unbound, so a caller that forgets to pass the roster gets no verdict rather than a silently permissive one, and an empty roster classifies as incomplete rather than passing every green rollup.
  • Verified: the derived roster for x45dev/firstmate is byte-identical to the constant it replaces, and all four affected pull requests now verify correctly, including workspace-template's six expanded matrix.include legs.

SECOND COMMIT, separately authorized and deliberately included in this same change: main is currently RED and it is not this change's doing. Commit 0dd2fc2 added a landing-target guard to bin/fm-pr-check.sh that resolves whether this machine can merge into the target repository and refuses to arm unless the verdict is mergeable or unchecked, failing closed on 'unreachable' by design. It did not update the pre-existing tests/fm-secondmate-safety.test.sh, whose FM_HOME parameterization case calls fm-pr-check.sh with the fixture URL https://github.com/example/repo/pull/1 purely to prove FM_HOME scopes data and state paths. No forge can resolve that repository, so the guard exits 1 and that test fails. Main was green at c8536c4 immediately before that merge and red at 75267ef immediately after.

This was fixed here, in its own commit, because a red main blocks every commit until it lands - that is the standing rule for a gate failure a change did not cause. The fix is TEST-ONLY and the boundary is explicit and must be held: do NOT weaken, bypass, or add a production escape hatch to the landing guard. Failing closed on a forge it cannot query is that guard working exactly as designed and is the entire reason it exists. The test's intent is FM_HOME path isolation and never landing authority, so the test stubs gh-axi on PATH to answer the repository permission read with one bare boolean - the same contract the existing fake in tests/fm-pr-check-security.test.sh reproduces, using the PATH=fakebin convention that test file already uses for tmux. The guard still runs its own resolve path rather than being bypassed. If any finding pushes toward relaxing the guard instead of the test, that must be escalated rather than applied.

Scope is deliberately limited to these two things: no refactors, no adjacent cleanups, no opportunistic improvements.

What Changed

  • fm_ci_roster (new, in bin/fm-ci-checks-lib.sh) resolves the required CI suite roster per repository by reading the job names of the newest successful CI run on the target branch, rather than reading them off the hardcoded FM_CI_REQUIRED_SUITES constant that previously listed firstmate's own twelve job names verbatim.
  • fm_ci_checks_state, fm_ci_run_jobs_state, and the underlying jq classifiers now take the roster as an explicit argument ($fm_ci_roster, bound via --argjson) instead of reading a shell constant; an unbound or empty roster fails closed to incomplete rather than passing every green rollup.
  • bin/fm-pr-ci-verify.sh resolves the roster from the PR's base repository and base branch before classifying, refuses with a clear error when no roster can be established, prints the roster's provenance alongside the verdict, and documents FM_CI_REQUIRED_SUITES as the override for a branch that deliberately adds or removes a CI job.
  • bin/fm-bearings-snapshot.sh resolves the roster once per repository (via its own bounded fm_ci_gh wrapper) before classifying that repository's PR rows, and CONTRIBUTING.md documents the override.
  • tests/fm-secondmate-safety.test.sh's FM_HOME parameterization case now stubs the gh-axi repository-permission read on PATH so the pre-existing landing-target guard in fm-pr-check.sh (added in a prior merge, unrelated to this change) doesn't fail the test on an unresolvable fixture repository; the guard itself is untouched.

Risk Assessment

✅ Low: The roster-resolution logic (fm_ci_roster), its fail-closed paths (unbound jq vars, empty-roster refusal, unpaginated-jobs refusal), and both call sites (fm-pr-ci-verify.sh resolving against the PR's base repo/branch, fm-bearings-snapshot.sh resolving once per repo) trace correctly against constructed inputs with no reachable false-green path found; all new tests exercise the real functions/scripts and assert observable behavior rather than grepping source; the second commit is verified test-only with the landing guard's production code and fail-closed behavior untouched, matching the explicit boundary in the intent.

Testing

Ran the two directly relevant test suites (fm-ci-checks, fm-secondmate-safety) to completion with no failures, reproduced both original bugs (hardcoded roster refusing a real go-bip39-validator PR, and the FM_HOME test failing on the landing guard) against pre-fix code to confirm they are genuine regressions this change addresses, then demonstrated the fix working live end-to-end against real GitHub repositories and PRs (not just fixtures) — the derived roster for firstmate is byte-identical to the old constant, and a genuinely different repo's PR now verifies correctly. fm-bearings-snapshot.test.sh's full run repeatedly stalled partway through on unrelated e2e/perl-timeout-fallback tests unconnected to this diff; this reproduces identically on the base commit and passes cleanly in isolation or small subsets (including the new roster-integration test test_include_prs_is_the_only_fetch_path), so it is pre-existing flakiness under this machine's current concurrent load rather than a regression from this change.

Evidence: fm-pr-ci-verify.sh on a real firstmate PR (roster resolves, refuses on real failing suite)
https://github.com/x45dev/firstmate/pull/2
required suites: 12, from x45dev/firstmate CI run 33228033272 on main
suite FAILURE Require no-mistakes / PR must be raised via no-mistakes
suite SUCCESS CI / Lint
...
suite FAILURE CI / Behavior portable serial 1
...
x45dev/firstmate checks: failing (13 repository-owned)
error: refusing to call ... its x45dev/firstmate checks are failing
EXIT=1
Evidence: Before/after on a real non-firstmate PR (x45dev/go-bip39-validator #4)
BASE COMMIT (old hardcoded roster):
suite SUCCESS CI / check
checks: incomplete (1 repository-owned)
missing required suites: Lint, Test coverage guard, ... (11 firstmate-only jobs)
EXIT=1

TARGET COMMIT (derived roster):
required suites: 1, from x45dev/go-bip39-validator CI run 33243632023 on main
suite SUCCESS CI / check
checks: passing (1 repository-owned)
validated: x45dev/go-bip39-validator suites passed on 91801176ea76217ffaf59b4173e22a5db575519c
EXIT=0
Evidence: Reproduced the FM_HOME test regression pre-fix, then confirmed fixed
Base-commit test against current fm-pr-check.sh: 'not ok - fm-pr-check failed under FM_HOME' (landing guard fails closed on unresolvable fixture repo). Target-commit test: 'ok - FM_HOME parameterizes data and state paths'.

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • bash tests/fm-ci-checks.test.sh — 52/52 passing, including the new fm_ci_roster unit tests and fm-pr-ci-verify.sh roster-integration tests
  • bash tests/fm-secondmate-safety.test.sh — full run, all passing, including test_fm_home_parameterization (the specific test the second commit fixed)
  • Reproduced the pre-fix regression: swapped in the base-commit tests/fm-secondmate-safety.test.sh against the current (unmodified) bin/fm-pr-check.sh — confirmed it fails with 'fm-pr-check failed under FM_HOME' because the landing guard fails closed on the unresolvable fixture repo, exactly as described in the intent; restored the file afterward
  • Live end-to-end: . bin/fm-ci-checks-lib.sh; fm_ci_roster x45dev/firstmate main against the real GitHub API — derived roster is byte-identical (sorted) to the old hardcoded FM_CI_REQUIRED_SUITES constant
  • Live end-to-end: bin/fm-pr-ci-verify.sh https://github.com/x45dev/firstmate/pull/2 — correctly resolves the 12-job roster and refuses on a real failing suite
  • Live end-to-end: fm_ci_roster x45dev/go-bip39-validator — derived roster is ["check"], nothing like firstmate's roster, proving the bug's root cause
  • Live end-to-end: bin/fm-pr-ci-verify.sh https://github.com/x45dev/go-bip39-validator/pull/4 (a real merged PR) — target commit accepts it as passing (exit 0); reproduced with the base-commit tool on the same PR to confirm it wrongly refused (exit 1, 'checks do not cover the required suite roster') before the fix
  • Live end-to-end: FM_CI_REQUIRED_SUITES override accepted with a valid array (exit 0) and refused closed with a malformed value (exit 1, no silent pass)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

x45dev added 2 commits August 30, 2026 10:14
FM_CI_REQUIRED_SUITES was a constant listing firstmate's own twelve CI job
names, so every other repository failed the completeness test by construction:
there is no roster to drift into on another repo, the roster simply does not
describe it. fm-pr-ci-verify.sh refused four consecutive real pull requests for
this reason, each independently hand-verified green, and fm-bearings-snapshot.sh
read every non-firstmate PR row as incomplete for the same reason.

fm_ci_roster now resolves the roster per repository and the classifiers take it
as an argument. jq refuses to compile a program whose variables are unbound, so
a caller that forgets the roster gets no verdict rather than a silent one, and
an empty roster classifies as incomplete rather than passing every green rollup.

Substitution flagged for review: the brief suggested reading job names from the
target repository's .github/workflows/ci.yml. That file is the definition but
not a roster, because a job name is a template GitHub evaluates ("Behavior
portable serial ${{ matrix.shard }}", or a matrix.include leg's
"${{ matrix.name }}"), so parsing it means reimplementing matrix expansion and
the Actions expression language. One of the three reported repositories needs
exactly that. The roster is therefore read by observation, from the job names of
the newest successful CI run on the branch the change targets, which GitHub has
already expanded exactly. No dependency is added: gh and jq only, both already
required. The cost is up to three GitHub API reads per repository per
verification. FM_CI_REQUIRED_SUITES survives as the documented override for a
change that deliberately adds or removes a CI job.

Verified: the derived roster for x45dev/firstmate is byte-identical to the
constant it replaces, and the three reported repositories plus the fourth field
case (home-fintech#76) now verify green, including workspace-template's six
expanded matrix.include legs.
Commit 0dd2fc2 added a landing-target guard to bin/fm-pr-check.sh without
updating this pre-existing test, and main has been red since that work merged at
75267ef. The guard resolves whether this machine can merge into the target
repository and refuses to arm unless the verdict is mergeable or unchecked,
failing closed on unreachable by design.

tests/fm-secondmate-safety.test.sh predates that guard. Its FM_HOME
parameterization case calls fm-pr-check.sh with the fixture URL
https://github.com/example/repo/pull/1 purely to prove that FM_HOME scopes data
and state paths. No forge can resolve that repository, so the guard now exits 1
and the assertion fails before reaching the path checks the case exists for. The
test needs a forge stub because its intent is FM_HOME path isolation and never
landing authority, and the guard is behaving exactly as designed, so the test is
what needed changing.

The stub is deliberately the smallest thing that removes the dependency: a
gh-axi on PATH answering the repository permission read with one bare boolean,
the same contract the fake in tests/fm-pr-check-security.test.sh reproduces, via
the PATH="$fakebin:$PATH" convention this file already uses for tmux. The guard
still runs its own resolve path rather than being bypassed, no escape hatch is
added to the library, and the case goes back to asserting path isolation.

Verified: the fixture command exits 1 with the refusal "could not confirm
example/repo is a landing path" without the stub and exits 0 with it, and the
suite is 40/40 with bin/fm-lint.sh clean.
@x45dev
x45dev merged commit 01da79b into main Aug 30, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant