fix(bin): resolve the gating workflow name per repository - #10
Merged
Merged
Conversation
bin/fm-ci-checks-lib.sh named the gating workflow with a constant, FM_CI_WORKFLOW_NAME='CI', and both classifiers filtered on it. On any repository whose gate is named anything else, every conclusion was discarded before the required-suite roster was consulted, so the command could only ever refuse. That is the roster constant's mistake one layer up: one repository's answer written down as every repository's. FM_CI_REQUIRED_SUITES could not fix it and no roster override can. The roster is bound separately as $fm_ci_roster and is read one layer below the workflow-name filter, so a rollup emptied by that filter never reaches it. The gate is now resolved per repository, following the shape of the roster fix and from the same query: the repository's own successful push runs on the target branch. The workflows among them are the gate, and the job names of the newest such run of each are the roster, so one API call answers both halves. Reading the target branch rather than the branch under test keeps a branch from certifying itself. Restricting to push runs is a real discriminator rather than a convenience: it excludes a dispatch-only release workflow that would otherwise drag its release jobs into every pull request's roster, and it excludes a pull_request-only policy workflow, which a fork validating a commit on its own branch push can never produce and which would therefore refuse the head-repository evidence this file exists to accept. Both halves are now jq variables the classifiers must be given, so a caller that forgets one gets no verdict rather than a silent misclassification. An unestablished gate reports incomplete, where every other unestablished standard already lands, so it can only ever cost a verdict. FM_CI_GATING_WORKFLOWS is the escape hatch for a repository the push-run rule gets wrong; set alone it still takes the roster from that workflow's observed runs, so naming a workflow the branch never validated is a refusal rather than a gate contributing no suites. The three outcomes stay distinct, in the portable tests and live: x45dev/agent-standards PR kunchenguid#110, gated by a workflow named lint, verifies green; a red suite and a gate that never ran are still refused; and a repository whose gate cannot be established still reports that it could not answer. docs/verification/ci-gate-resolution.md records the live evidence the stubbed tests cannot carry. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UtpcpfKkD8k3AW9sK7YK51
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Make the gating workflow name resolvable per repository, so bin/fm-pr-ci-verify.sh can answer on a repository whose workflow is not literally named "CI".
The defect: bin/fm-ci-checks-lib.sh set FM_CI_WORKFLOW_NAME='CI' as a plain assignment with no environment fallback, and fm_ci_from_ci_workflow filtered conclusions on that literal name. Any repository whose gating workflow is named anything else had every conclusion discarded before the required-suite roster was consulted, and the command reported state=no-repo-ci.
FM_CI_REQUIRED_SUITES does not and cannot fix this, and was explicitly ruled out. Two separate workers proposed it on 2026-09-01 and 2026-09-02 and it was wrong both times. The roster is bound separately through --argjson as $fm_ci_roster and never reaches the workflow-name filter. The roster fixes job names one layer below this defect; that fix already landed as #7 and is the structural model for this change, not its solution.
Evidence, verified independently on two separate days: x45dev/agent-standards has two workflows. Only lint.yml carries a pull_request trigger, and it declares name: lint; tag-release.yml is workflow_dispatch only, so that one check is the whole pull request gate for that repository. gh-axi pr checks reports it passing, and the verifier still refused. This was hit on PR kunchenguid#104 on 2026-09-01, and again on PRs kunchenguid#108, kunchenguid#109 and kunchenguid#110 on 2026-09-02, where the green result had to be established by hand each time.
What must survive the change: the command distinguishes three outcomes - verified green, verified red, and could not verify. All three must stay distinct. Collapsing "could not verify" into either of the others is the actual danger here, and is worse than the bug being fixed: AGENTS.md requires confirming a green claim with this command and never taking one on trust, so a verifier that says green when it could not tell defeats the rule it exists to serve. The command's own purpose must not be weakened: an all-green check list is not evidence the suites ran, because a pull request can carry only a third-party bot's pass while every repository suite is held unapproved, which reads as "1 passed, 0 failed" to anything that merely counts conclusions. Whatever resolution mechanism is built must still be able to tell those apart.
Scope: per-repository resolution of the gating workflow name, following the shape of the already-landed roster fix. Not to be widened into a rework of the verifier.
Required proof, demonstrated rather than asserted: (1) a repository whose gating workflow is not named CI now verifies green when it is green, using x45dev/agent-standards and a real merged pull request from 2026-09-02; (2) a genuinely red or unrun state still reports as such rather than as green; (3) a case the command genuinely cannot answer still reports that it could not answer.
This is firstmate's own shared tracked code, so the firstmate-coding-guidelines skill governs it: knowledge-placement decision tree, the one-owner rule for contracts, one sentence per line in tracked Markdown, plain dash and never an em dash, no agent co-author, shellcheck-clean bin scripts, bin/fm-lint.sh as the single owner of the lint definition, tests colocated in tests/ extending the existing runner, tests exercising behavior through an executable interface and never asserting implementation-source bytes, and maintainer-verification records under docs/verification/ carrying dated exact commands and output.
Decisions and tradeoffs made while doing the work, which a reviewer reading only the diff would not know:
Delivery: this task's recorded posture is no-mistakes with merge approval reserved for the captain, which overrides this repository's usual standing merge authority. Open the pull request, report it, and stop without merging.
What Changed
bin/fm-ci-checks-lib.sh: replaced the hardcodedFM_CI_WORKFLOW_NAME='CI'filter with per-repository resolution of the set of gating workflow names, derived by observation from the target branch's own successful push runs (the same query the required-suite roster is read from), withFM_CI_GATING_WORKFLOWSadded as an escape hatch override; classifiers now take both$fm_ci_workflowsand$fm_ci_rosteras required jq variables so an unbound one fails the call instead of silently misclassifying; an unestablished gate now reportsincompleteinstead ofno-repo-ci; fixed a single-quoted string terminated early by an apostrophe in a jq comment, and a jqindex()binding order bug onworkflowName.bin/fm-pr-ci-verify.shandbin/fm-bearings-snapshot.sh: updated all inline jq call sites to bind and pass through the new$fm_ci_workflowsvariable alongside the existing roster binding.tests/fm-ci-checks.test.shandtests/fm-bearings-snapshot.test.sh: extended coverage for the new resolution behavior, theincompleteoutcome, and the escape-hatch override.docs/verification/ci-gate-resolution.md(new) anddocs/documentation-audiences.json: added a maintainer-verification record with dated live-repository evidence (x45dev/agent-standards, x45dev/firstmate, octocat/Hello-World) demonstrating the three required outcomes stay distinct.CONTRIBUTING.md: minor doc update accompanying the change.Risk Assessment
✅ Low: The change is well-bounded: it resolves the gate and roster from one observed query, preserves all three required outcomes (verified via new incomplete/no-gate handling and matching tests), threads the new $fm_ci_workflows binding through every jq call site (verified none were missed), keeps shellcheck clean, and the live verification doc demonstrates all three required proof scenarios against real repositories.
Testing
Ran the two changed portable test files (fm-ci-checks.test.sh: 58/58, fm-bearings-snapshot.test.sh: 41/41, both exit 0), then independently reproduced all three required live proofs from docs/verification/ci-gate-resolution.md against the real GitHub API rather than trusting the doc: x45dev/agent-standards PR kunchenguid#110 (gate named lint) verifies green (exit 0); x45dev/firstmate PR #4 (one genuinely red suite among 25) still refuses as failing (exit 1); x45dev/firstmate PR #1 (gate never ran) still refuses as no-repo-ci (exit 1); and octocat/Hello-World PR #11064 (no push-run gate resolvable) reports could-not-verify with distinct wording and never prints validated: (exit 1). All outputs matched the maintainer-verification doc exactly, confirming the three outcomes (verified green, verified red/unrun, could not verify) remain distinct end to end on repositories whose gating workflow is not literally named CI. No issues found; working tree left clean.
Evidence: Live proof (1): repo with non-CI-named gate (workflow "lint") verifies green
Evidence: Live proof (2): genuinely red suite and never-run gate both still refuse
Evidence: Live proof (3): repository with no resolvable gate reports "could not verify", never green
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
bash tests/fm-ci-checks.test.sh (58/58 pass, exit 0) - unit coverage of fm_ci_roster, fm_ci_checks_state, fm_ci_runs_state, fm_ci_run_jobs_state and the fm-pr-ci-verify.sh guard against stubbed ghbash tests/fm-bearings-snapshot.test.sh (41/41 pass, exit 0)Live:bin/fm-pr-ci-verify.sh https://github.com/x45dev/agent-standards/pull/110- a repository gated by a workflow namedlint(notCI) verifies green, exit 0Live:bin/fm-pr-ci-verify.sh https://github.com/x45dev/firstmate/pull/4- a real red suite (CI / Behavior portable serial 4) refuses, exit 1Live:bin/fm-pr-ci-verify.sh https://github.com/x45dev/firstmate/pull/1- a gate that never ran on the commit refuses as no-repo-ci, exit 1Live:bin/fm-pr-ci-verify.sh https://github.com/octocat/Hello-World/pull/11064- a repository whose gate cannot be established reports could-not-verify (distinct error text, nevervalidated:), exit 1gh api 'repos/x45dev/agent-standards/actions/workflows?per_page=100'and the branch-runs query - confirmed live that agent-standards ownslint(push-triggered) andtag-release(workflow_dispatch only, correctly excluded from the gate)git status --short - working tree clean, no test artifacts left behind✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.