Skip to content

fix(bin): resolve the gating workflow name per repository - #10

Merged
x45dev merged 1 commit into
mainfrom
fm/fm-ci-verify-workflow-name-hardcoded
Sep 2, 2026
Merged

x45dev merged 1 commit into
mainfrom
fm/fm-ci-verify-workflow-name-hardcoded

Conversation

@x45dev

@x45dev x45dev commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Intent

Make the gating workflow name resolvable per repository, so bin/fm-pr-ci-verify.sh can answer on a repository whose workflow is not literally named "CI".

The defect: bin/fm-ci-checks-lib.sh set FM_CI_WORKFLOW_NAME='CI' as a plain assignment with no environment fallback, and fm_ci_from_ci_workflow filtered conclusions on that literal name. Any repository whose gating workflow is named anything else had every conclusion discarded before the required-suite roster was consulted, and the command reported state=no-repo-ci.

FM_CI_REQUIRED_SUITES does not and cannot fix this, and was explicitly ruled out. Two separate workers proposed it on 2026-09-01 and 2026-09-02 and it was wrong both times. The roster is bound separately through --argjson as $fm_ci_roster and never reaches the workflow-name filter. The roster fixes job names one layer below this defect; that fix already landed as #7 and is the structural model for this change, not its solution.

Evidence, verified independently on two separate days: x45dev/agent-standards has two workflows. Only lint.yml carries a pull_request trigger, and it declares name: lint; tag-release.yml is workflow_dispatch only, so that one check is the whole pull request gate for that repository. gh-axi pr checks reports it passing, and the verifier still refused. This was hit on PR kunchenguid#104 on 2026-09-01, and again on PRs kunchenguid#108, kunchenguid#109 and kunchenguid#110 on 2026-09-02, where the green result had to be established by hand each time.

What must survive the change: the command distinguishes three outcomes - verified green, verified red, and could not verify. All three must stay distinct. Collapsing "could not verify" into either of the others is the actual danger here, and is worse than the bug being fixed: AGENTS.md requires confirming a green claim with this command and never taking one on trust, so a verifier that says green when it could not tell defeats the rule it exists to serve. The command's own purpose must not be weakened: an all-green check list is not evidence the suites ran, because a pull request can carry only a third-party bot's pass while every repository suite is held unapproved, which reads as "1 passed, 0 failed" to anything that merely counts conclusions. Whatever resolution mechanism is built must still be able to tell those apart.

Scope: per-repository resolution of the gating workflow name, following the shape of the already-landed roster fix. Not to be widened into a rework of the verifier.

Required proof, demonstrated rather than asserted: (1) a repository whose gating workflow is not named CI now verifies green when it is green, using x45dev/agent-standards and a real merged pull request from 2026-09-02; (2) a genuinely red or unrun state still reports as such rather than as green; (3) a case the command genuinely cannot answer still reports that it could not answer.

This is firstmate's own shared tracked code, so the firstmate-coding-guidelines skill governs it: knowledge-placement decision tree, the one-owner rule for contracts, one sentence per line in tracked Markdown, plain dash and never an em dash, no agent co-author, shellcheck-clean bin scripts, bin/fm-lint.sh as the single owner of the lint definition, tests colocated in tests/ extending the existing runner, tests exercising behavior through an executable interface and never asserting implementation-source bytes, and maintainer-verification records under docs/verification/ carrying dated exact commands and output.

Decisions and tradeoffs made while doing the work, which a reviewer reading only the diff would not know:

  • The gate is resolved by OBSERVATION from the same query the roster is read from: the repository's own successful PUSH runs on the TARGET branch, one API call answering both halves (which workflows gate the repository, and the job names of the newest such run of each as the roster). This follows the roster fix's shape deliberately: GitHub has already evaluated the workflow definitions in every run it recorded, so parsing .github/workflows/ was rejected for the same reason the roster rejected it.
  • Reading the target branch rather than the branch under test is deliberate: a branch that deleted or thinned the gating workflow would otherwise certify itself.
  • Restricting to push runs is a real discriminator rather than a convenience, and both directions were checked against live repositories. It excludes x45dev/agent-standards' tag-release, which is workflow_dispatch only and would otherwise drag its release jobs into every pull request's roster. It also excludes firstmate's own "Require no-mistakes", which is pull_request-only: a fork validating a commit on its own branch push can never produce such a run, so including it would refuse the head-repository evidence this file exists to accept. Resolving the gate as "every workflow that triggers on pull_request" was considered and rejected for exactly that reason.
  • The gate is a SET of workflow names, not a single name, even though the task says "name". Resolving one name and refusing on ambiguity would leave the command refusing on any ordinary repository with two push-validated workflows, and requiring more workflows is strictly stricter, never looser. This was judged the honest generalization rather than a widening: the verifier's state machine, its three outcomes and its two evidence shapes are untouched.
  • Both halves are now jq variables the classifiers must be given ($fm_ci_workflows alongside $fm_ci_roster), because jq refuses to compile a program whose variables are unbound, so a caller that forgets one gets no verdict at all rather than a silent misclassification. This is why every inline jq call site in fm-pr-ci-verify.sh and fm-bearings-snapshot.sh gained the binding.
  • An unestablished gate reports "incomplete", not "no-repo-ci": with no names to match every check reads as belonging to somebody else, and no-repo-ci is a finding about the pull request when what is actually missing is the standard. incomplete is where every other unestablished standard already lands, and it preserves bearings' existing behaviour for a repository whose standard cannot be established.
  • fm_ci_no_gate is decided after "none" and before "no-repo-ci" in the rollup shape, deliberately preserving the existing rule that no-repo-ci is decided before red and pending.
  • FM_CI_GATING_WORKFLOWS was added as the escape hatch, mirroring FM_CI_REQUIRED_SUITES. Set alone it still takes the roster from those workflows' observed runs, so naming a workflow the branch never validated is a refusal rather than a workflow silently contributing no suites. Only both overrides together reach no network at all.
  • A knowingly accepted boundary, documented in the library header rather than hidden: a workflow that runs on a push to the target branch but not on pull requests (a deploy or release-notes job) lands in the gate, so a pull request carrying no run of it reads incomplete. That costs a verdict rather than granting one, which is the correct direction for this command, and the override covers it. The 100-run window on the resolution query is the other accepted bound, taken in exchange for answering in one API call.
  • One shell hazard was found and fixed during the work: an apostrophe in a comment inside the single-quoted FM_CI_CHECKS_JQ_DEFS string terminates that string. The comment was reworded rather than escaped.
  • jq's index() argument is evaluated with . bound to the left of the pipe, so the check run's own workflowName is captured with "as $w" before the membership test; the naive form raised "Cannot index array with string".
  • docs/verification/ci-gate-resolution.md was added because the resolution's core claim is a live, network-dependent empirical fact that the stubbed portable tests cannot carry, following the existing pattern of docs/verification/pr-landing-target.md. It is registered in docs/documentation-audiences.json as maintainer-verification.
  • The three required proofs were demonstrated live before committing: x45dev/agent-standards PRs feat(bin): orchestrator dashboard layout with read-only worker watch panes kunchenguid/firstmate#110, feat(supervise): add deterministic crew state helper kunchenguid/firstmate#104, fix(watch): add durable active watcher session kunchenguid/firstmate#108 and fix: clarify X-mode owner mention handling kunchenguid/firstmate#109 all verify green (exit 0); x45dev/firstmate PR fix(bin): refuse an unmergeable upstream pull request as a landing path #4 with one red suite still refuses as failing (exit 1); x45dev/firstmate PR fix(bin): accept effort on opencode/kimi profiles as documentation #1 whose gate never ran still refuses (exit 1); and octocat/Hello-World PR #11064, a repository that has never run a workflow on a push to its default branch, still reports that the gate could not be established (exit 1), including when a gate is named for it by hand.

Delivery: this task's recorded posture is no-mistakes with merge approval reserved for the captain, which overrides this repository's usual standing merge authority. Open the pull request, report it, and stop without merging.

What Changed

  • bin/fm-ci-checks-lib.sh: replaced the hardcoded FM_CI_WORKFLOW_NAME='CI' filter with per-repository resolution of the set of gating workflow names, derived by observation from the target branch's own successful push runs (the same query the required-suite roster is read from), with FM_CI_GATING_WORKFLOWS added as an escape hatch override; classifiers now take both $fm_ci_workflows and $fm_ci_roster as required jq variables so an unbound one fails the call instead of silently misclassifying; an unestablished gate now reports incomplete instead of no-repo-ci; fixed a single-quoted string terminated early by an apostrophe in a jq comment, and a jq index() binding order bug on workflowName.
  • bin/fm-pr-ci-verify.sh and bin/fm-bearings-snapshot.sh: updated all inline jq call sites to bind and pass through the new $fm_ci_workflows variable alongside the existing roster binding.
  • tests/fm-ci-checks.test.sh and tests/fm-bearings-snapshot.test.sh: extended coverage for the new resolution behavior, the incomplete outcome, and the escape-hatch override.
  • docs/verification/ci-gate-resolution.md (new) and docs/documentation-audiences.json: added a maintainer-verification record with dated live-repository evidence (x45dev/agent-standards, x45dev/firstmate, octocat/Hello-World) demonstrating the three required outcomes stay distinct.
  • CONTRIBUTING.md: minor doc update accompanying the change.

Risk Assessment

✅ Low: The change is well-bounded: it resolves the gate and roster from one observed query, preserves all three required outcomes (verified via new incomplete/no-gate handling and matching tests), threads the new $fm_ci_workflows binding through every jq call site (verified none were missed), keeps shellcheck clean, and the live verification doc demonstrates all three required proof scenarios against real repositories.

Testing

Ran the two changed portable test files (fm-ci-checks.test.sh: 58/58, fm-bearings-snapshot.test.sh: 41/41, both exit 0), then independently reproduced all three required live proofs from docs/verification/ci-gate-resolution.md against the real GitHub API rather than trusting the doc: x45dev/agent-standards PR kunchenguid#110 (gate named lint) verifies green (exit 0); x45dev/firstmate PR #4 (one genuinely red suite among 25) still refuses as failing (exit 1); x45dev/firstmate PR #1 (gate never ran) still refuses as no-repo-ci (exit 1); and octocat/Hello-World PR #11064 (no push-run gate resolvable) reports could-not-verify with distinct wording and never prints validated: (exit 1). All outputs matched the maintainer-verification doc exactly, confirming the three outcomes (verified green, verified red/unrun, could not verify) remain distinct end to end on repositories whose gating workflow is not literally named CI. No issues found; working tree left clean.

Evidence: Live proof (1): repo with non-CI-named gate (workflow "lint") verifies green
$ bin/fm-pr-ci-verify.sh https://github.com/x45dev/agent-standards/pull/110
gating workflows: lint, from x45dev/agent-standards successful push runs on main
required suites: 1, from x45dev/agent-standards lint run 33598952234 on main
suite SUCCESS lint / lint
x45dev/agent-standards checks: passing (1 repository-owned)
validated: x45dev/agent-standards suites passed on 76291ea2e6bdb2778f308d05a4ccb1baa9a6d555 in x45dev/agent-standards
$ echo $?
0
Evidence: Live proof (2): genuinely red suite and never-run gate both still refuse
$ bin/fm-pr-ci-verify.sh https://github.com/x45dev/firstmate/pull/4
... (25 repository-owned suites, one FAILURE: "CI / Behavior portable serial 4")
x45dev/firstmate checks: failing (25 repository-owned)
error: refusing to call https://github.com/x45dev/firstmate/pull/4 green: its x45dev/firstmate checks are failing (see the roster above).
$ echo $?
1

$ bin/fm-pr-ci-verify.sh https://github.com/x45dev/firstmate/pull/1
x45dev/firstmate checks: none (0 repository-owned)
error: refusing to call https://github.com/x45dev/firstmate/pull/1 green: no x45dev/firstmate suite ran on this commit.
$ echo $?
1
Evidence: Live proof (3): repository with no resolvable gate reports "could not verify", never green
$ bin/fm-pr-ci-verify.sh https://github.com/octocat/Hello-World/pull/11064
error: octocat/Hello-World has run no workflow on a push to master, so its gating workflows cannot be established
error: refusing to call https://github.com/octocat/Hello-World/pull/11064 green: could not establish what octocat/Hello-World requires of a commit.
$ echo $?
1

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • bash tests/fm-ci-checks.test.sh (58/58 pass, exit 0) - unit coverage of fm_ci_roster, fm_ci_checks_state, fm_ci_runs_state, fm_ci_run_jobs_state and the fm-pr-ci-verify.sh guard against stubbed gh
  • bash tests/fm-bearings-snapshot.test.sh (41/41 pass, exit 0)
  • Live: bin/fm-pr-ci-verify.sh https://github.com/x45dev/agent-standards/pull/110 - a repository gated by a workflow named lint (not CI) verifies green, exit 0
  • Live: bin/fm-pr-ci-verify.sh https://github.com/x45dev/firstmate/pull/4 - a real red suite (CI / Behavior portable serial 4) refuses, exit 1
  • Live: bin/fm-pr-ci-verify.sh https://github.com/x45dev/firstmate/pull/1 - a gate that never ran on the commit refuses as no-repo-ci, exit 1
  • Live: bin/fm-pr-ci-verify.sh https://github.com/octocat/Hello-World/pull/11064 - a repository whose gate cannot be established reports could-not-verify (distinct error text, never validated:), exit 1
  • gh api 'repos/x45dev/agent-standards/actions/workflows?per_page=100' and the branch-runs query - confirmed live that agent-standards owns lint (push-triggered) and tag-release (workflow_dispatch only, correctly excluded from the gate)
  • git status --short - working tree clean, no test artifacts left behind
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

bin/fm-ci-checks-lib.sh named the gating workflow with a constant,
FM_CI_WORKFLOW_NAME='CI', and both classifiers filtered on it. On any
repository whose gate is named anything else, every conclusion was
discarded before the required-suite roster was consulted, so the command
could only ever refuse. That is the roster constant's mistake one layer
up: one repository's answer written down as every repository's.

FM_CI_REQUIRED_SUITES could not fix it and no roster override can. The
roster is bound separately as $fm_ci_roster and is read one layer below
the workflow-name filter, so a rollup emptied by that filter never
reaches it.

The gate is now resolved per repository, following the shape of the
roster fix and from the same query: the repository's own successful push
runs on the target branch. The workflows among them are the gate, and
the job names of the newest such run of each are the roster, so one API
call answers both halves. Reading the target branch rather than the
branch under test keeps a branch from certifying itself. Restricting to
push runs is a real discriminator rather than a convenience: it excludes
a dispatch-only release workflow that would otherwise drag its release
jobs into every pull request's roster, and it excludes a pull_request-only
policy workflow, which a fork validating a commit on its own branch push
can never produce and which would therefore refuse the head-repository
evidence this file exists to accept.

Both halves are now jq variables the classifiers must be given, so a
caller that forgets one gets no verdict rather than a silent
misclassification. An unestablished gate reports incomplete, where every
other unestablished standard already lands, so it can only ever cost a
verdict. FM_CI_GATING_WORKFLOWS is the escape hatch for a repository the
push-run rule gets wrong; set alone it still takes the roster from that
workflow's observed runs, so naming a workflow the branch never
validated is a refusal rather than a gate contributing no suites.

The three outcomes stay distinct, in the portable tests and live:
x45dev/agent-standards PR kunchenguid#110, gated by a workflow named lint, verifies
green; a red suite and a gate that never ran are still refused; and a
repository whose gate cannot be established still reports that it could
not answer. docs/verification/ci-gate-resolution.md records the live
evidence the stubbed tests cannot carry.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UtpcpfKkD8k3AW9sK7YK51
@x45dev
x45dev merged commit 3900447 into main Sep 2, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant