diff --git a/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md b/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md index d0fd075cc9..73dbdc2d10 100644 --- a/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md +++ b/.agents/skills/nemoclaw-maintainer-day/MERGE-GATE.md @@ -10,9 +10,9 @@ Run the last maintainer check before approval. Never merge automatically. For the full priority list see [PR-REVIEW-PRIORITIES.md](PR-REVIEW-PRIORITIES.md). A PR is approval-ready only when **all** hard gates pass: 1. **Product scope approved** — confirm that the PR implements existing supported behavior or a linked, accepted product decision. Do not approve a new integration, solution, third-party stack, custom image, or canonical documentation surface merely because it works. Require defined ownership, lifecycle, compatibility, security, and validation expectations. Route independent solutions through [Community Solutions](../../../docs/resources/community-contributions.mdx). -2. **Contributor compliance** — the PR body contains the contributor's `Signed-off-by:` declaration and every PR commit appears as `Verified` in GitHub. Reject noncompliant PRs; maintainers do not repair contributor history. -3. **CI green** — all required checks in `statusCheckRollup`. -4. **No conflicts** — `mergeStateStatus` clean. +2. **Contributor compliance** — the PR body contains the contributor's `Signed-off-by:` declaration and every PR commit appears as `Verified` in GitHub. Only PR authors whose case-normalized login is exactly `dependabot[bot]` or `app/dependabot` are exempt from the PR-body declaration; every Dependabot commit must still be verified. Reject other noncompliant PRs; maintainers do not repair contributor history. +3. **CI green on the exact diff** — every required check has successful, complete evidence bound to the captured PR head and base. +4. **Current merge revision** — the PR remains open and non-draft; its title, body, head, base branch, base revision, mergeability, and merge state do not change during evaluation; the captured base is still current; and GitHub reports `MERGEABLE` with an allowed merge state. 5. **No major CodeRabbit** — ignore style nits; block on correctness/security bugs. 6. **Risky code tested** — see [RISKY-AREAS.md](RISKY-AREAS.md). Confirm tests exist (added or pre-existing). @@ -31,6 +31,7 @@ The script handles the deterministic checks. You handle judgment calls: - **Product scope not established:** Stop before approval when the PR would create a new supported product surface and no accepted issue or design decision establishes ownership and lifecycle expectations. Technical correctness, successful tests, green CI, and positive advisor output do not substitute for product approval. Ask a maintainer for the product decision or route an independent solution through [Community Solutions](../../../docs/resources/community-contributions.mdx). - **Missing required checks:** The checked-in script requires `checks`, `check-hash`, `changes`, `commit-lint`, `dco-check`, and `E2E / PR Gate` in the status rollup, including while the repository ruleset rollout is staged. First-time fork contributors may need "Approve and run" before the ordinary `pull_request` checks appear. The trusted E2E controller records the head and base for internal and fork PRs without executing PR code. Never waive a missing, neutral, or skipped E2E gate, and never run untrusted fork code with privileged credentials to manufacture a result. +- **Immutable Actions evidence:** Required PR workflows must identify the exact PR, head, and base. The installer-hash workflow reruns its trusted verification on every `pull_request` `edited` event, including non-base metadata edits, and records `gate true` with the current base in its immutable run name; edits never mint skipped required-check evidence. Every substantive `pull_request_target` E2E controller run likewise requires an immutable `gate true` run name for the exact PR, head, and base. GitHub's PR association normally provides the remaining run binding. When GitHub returns an empty association, the checker additionally requires the exact controller path, head branch, and head repository, and a run time that encloses the trusted exact coordination check. A later all-skipped `gate false` metadata run may be discarded using that same narrow relationship only when the stronger substantive run exists. Missing, malformed, stale, contradictory, or mutated title, coordination, timestamp, branch, repository, head, or base evidence fails closed. - **Controller workflow versus PR verdict:** `E2E / PR Gate Controller` reports whether the trusted controller safely published an outcome. It can be green while the custom `E2E / PR Gate` check is failed or cancelled. Merge authority comes from the custom check for the current head and base, not from the controller workflow. `Superseded by PR update` and `PR closed — gate no longer applies` are cancelled outcomes for an obsolete revision and require no action on that old check; the current revision still needs its own successful gate. The closed-PR outcome also covers a deleted fork repository whose GitHub head-repository field is empty. - **CI or selected E2E did not pass:** A handled prerequisite-CI or selected-E2E failure leaves `E2E / PR Gate` red while `E2E / PR Gate Controller` completes successfully. Follow the CI run, E2E run, non-passing job, and failed-step links in the custom check, fix or rerun the failing work, and require a successful gate for the resulting revision. A green controller run does not override the red verdict. - **Evidence could not be verified:** If a selected child succeeds but `Download evidence` fails, is cancelled, or is skipped, the custom check fails closed with `Evidence could not be verified` and `E2E / PR Gate Controller` is red. Inspect the `Download evidence` step and rerun the gate; the green child alone is not accepted evidence. A successful download with missing, duplicate, skipped, pending, or reported-failing signals is a handled red PR verdict with a green controller. Malformed or unsafe evidence, schema or identity mismatches, and traversal-limit violations remain controller verification errors: both the check and controller fail closed. @@ -38,11 +39,11 @@ The script handles the deterministic checks. You handle judgment calls: - **Manual fork PR credentialed E2E skip:** Use this only when the gate has failed with `Maintainer approval required to skip credentialed E2E`. Independently review the fork change and non-secret PR CI, then prefer the protected-environment approval above. The controller rejects a closed PR, a changed head or base, an internal PR, an empty E2E plan, a non-failed gate, a controller commit that is neither current `main` nor a safely validated ancestor of it, or a recorded reviewer below maintainer role. A safe `main` advance must preserve the controller commit as its merge base, contain fewer than 300 fully enumerated changed files, and avoid `e2e-control-plane` changes; other advances fail closed. It re-reads the open PR's head and base immediately before writing success. The result records the reviewer, optional bounded comment, validated approval-run URL, plan, and jobs that did not run. For the fallback, choose **Run workflow** on `main`, select `approve-fork-e2e-skip`, and provide the fork PR number, its current 40-character head SHA as `expected_head_sha`, its current 40-character base SHA as `expected_base_sha`, and a specific 10–500-character reason. Re-read both PR SHAs immediately before dispatch. Optionally set `evidence_url` to a reviewed URL matching `https://github.com/NVIDIA/NemoClaw/actions/runs/`, or leave it blank when no supporting run exists. PR, issue, comment, job, and external URLs are rejected. The controller validates the optional URL's repository and run-ID shape but does not inspect that run's head, jobs, or conclusion. - **E2E control-plane authorization:** The deterministic `e2e-control-plane` family remains a conservative path boundary covering E2E and PR-CI workflows, risk policy, dependency and test configuration, preparation and upload actions, and non-documentation files under `tools/e2e/` and `test/e2e/`, including shell and Python support files. Internal PRs whose matched control-plane files are drawn only from `.github/workflows/pr-e2e-gate.yaml` and `tools/e2e/pr-e2e-gate.mts` automatically dispatch the selected jobs for that SHA. Any other or mixed internal control-plane diff fails with `Maintainer authorization required to run E2E` without running selected jobs or exposing repository secrets. Independently review the revision and its non-secret CI, then choose **Run workflow** on `main`, select `run-control-plane`, and provide the PR number, current 40-character `expected_head_sha`, current 40-character `expected_base_sha`, and a specific 10–500-character `review_reason`. Re-read both SHAs immediately before dispatch. The first-attempt workflow requires the triggering actor to have `maintain` or `admin`, and rejects forks, stale or closed PRs, plans that do not require authorization, empty selections, a missing or mismatched failed gate, and an incompatible trusted controller commit. It re-reads the current head and base before dispatch. Authorization returns the existing check to in progress and runs the selected jobs through the normal wait, evidence-download, and finish path; it cannot directly record success. Only verified evidence for that SHA can make `E2E / PR Gate` green. - **Two-phase ruleset rollout and backfill:** First deploy the E2E check producer and its trusted fork handling. Re-run `CI / Pull Request` for every already-open PR head/base pair (approving a first-time fork run when necessary), then verify that `E2E / PR Gate` is attached to that same head SHA for the current base. The fail-closed maintainer checker identifies any head still needing backfill. Adding the workflow does not retroactively report the context, and enabling the ruleset first leaves existing PRs at "Waiting for status to be reported." Do not activate the context as a security boundary while it is scoped only to the shared GitHub Actions app: that app identity cannot distinguish this workflow from another workflow. First give the gate distinct provenance through a dedicated GitHub App or an organization required-workflow rule. When the context is finally activated, also enable strict/up-to-date required status checks; otherwise a successful head can remain mergeable after `main` advances and changes the effective merge diff. Preserve the control-plane review and deterministic floor even after those external protections are in place. -- **Contributor compliance failed:** Reject the PR and ask the contributor to provide the PR-body DCO declaration or replace unverified commits with a clean verified history. Do not approve, merge, amend, sign, or force-push on the contributor's behalf. +- **Contributor compliance failed:** Reject the PR and ask the contributor to provide the PR-body DCO declaration or replace unverified commits with a clean verified history. The only PR-body declaration exemption is the two exact Dependabot logins above; it does not exempt commit verification. Do not approve, merge, amend, sign, or force-push on the contributor's behalf. - **Contributor/approver overlap:** Surface `advisories.contributorApprovalOverlap` when the same account not recognized as automated by the supported login conventions appears as the current PR opener, commit author, or co-author and its latest opinionated review is approved. The invalid state detected here is contributor and approver identity overlap in the current GitHub PR metadata; the source boundary is the current opener plus all commit-author and review pages fetched through GitHub's GraphQL API. The advisory includes contributors whose commits remain in the current PR head at check time; it does not retain original push actors or authors removed when history is rebased, squashed, or fixed up. A clear result is not proof of independent approval. Missing, invalid, or conflicting review timestamps, or failure to retrieve complete paginated history, produce a warning because the latest opinion cannot be selected reliably. This is intentionally diagnostic-only under the maintainer scope decision recorded in the #6233 discussion; #6222 remains the broader proposal context. It is not an independent-approval policy, required check, branch-protection rule, or substitute for explicit human merge authorization, so it does not invalidate approval, require another reviewer, or change `allPass` or merge readiness. Mocked-GitHub regression tests cover opener and commit-author/co-author overlap, bot filtering, case normalization, latest-review transitions across API pages, timestamp ordering, incomplete timestamps, and incomplete paginated history. Remove this advisory if GitHub or a maintainer-approved authoritative control provides the same overlap signal, or replace it if the project adopts an enforced independent-approval policy. -- **Conflicts (DIRTY):** Do NOT approve — GitHub invalidates approvals when new commits are pushed. Salvage first (rebase), wait for CI, then re-run the gate checker. Follow [SALVAGE-PR.md](SALVAGE-PR.md). +- **Stale or changed revision:** Do NOT approve when the branch is behind, GitHub does not report `MERGEABLE`, the merge state is not allowed, or the PR head, base, state, draft status, mergeability, or merge state changes during evaluation. Refresh the branch when needed, wait for CI, then re-run the gate checker. Follow [SALVAGE-PR.md](SALVAGE-PR.md). - **CI failing but narrow:** Follow the salvage workflow in [SALVAGE-PR.md](SALVAGE-PR.md). - **CI pending:** Wait and re-check. Do not approve while checks are still running. - **CodeRabbit:** Script flags unresolved major/critical threads. Review the `snippet` to confirm it's a real issue vs style nit. If doubt, leave unapproved. @@ -51,7 +52,7 @@ The script handles the deterministic checks. You handle judgment calls: ## Step 3: Approve or Report -**Approve only when:** the human product-scope gate passes, `allPass` is true, `mergeStateStatus` is not DIRTY, and maintainer review found no unresolved correctness or security issue. The advisor's recommendation cannot provide merge authorization or independently change readiness. Approving a PR with conflicts is wasted effort — the rebase will invalidate the approval. +**Approve only when:** the human product-scope gate passes, `allPass` is true for the final unchanged revision, GitHub reports `MERGEABLE` with an allowed current-base merge state, and maintainer review found no unresolved correctness or security issue. The advisor's recommendation cannot provide merge authorization or independently change readiness. Approving a stale or conflicted PR is wasted effort — refreshing it will invalidate the approval. The correct sequence for a conflicted PR: **salvage (rebase) → CI green → approve → report ready for merge.** @@ -66,6 +67,6 @@ If the contributor/approver advisory is present, include it in the summary witho | Gate | Status | What is needed | |------|--------|----------------| | CI | Failing | Fix flaky timeout test | -| Conflicts | DIRTY | Rebase onto main first — approval would be invalidated | +| Conflicts | Not `MERGEABLE` or merge state not allowed/current | Rebase onto main first — approval would be invalidated | Use full GitHub links. diff --git a/.agents/skills/nemoclaw-maintainer-day/scripts/check-gates.ts b/.agents/skills/nemoclaw-maintainer-day/scripts/check-gates.ts index 86216d1481..6564ef18d5 100644 --- a/.agents/skills/nemoclaw-maintainer-day/scripts/check-gates.ts +++ b/.agents/skills/nemoclaw-maintainer-day/scripts/check-gates.ts @@ -74,11 +74,17 @@ interface GateOutput { pendingChecks?: string[]; missingChecks?: string[]; }; - conflicts: GateResult & { mergeStateStatus?: string }; + conflicts: GateResult & { + mergeable?: string; + mergeStateStatus?: string; + baseSha?: string; + currentBaseSha?: string; + }; coderabbit: GateResult & { unresolvedThreads?: CodeRabbitThread[] }; riskyCodeTested: GateResult & { riskyFiles?: string[]; hasTests?: boolean }; contributorCompliance: GateResult & { dcoDeclarationPresent?: boolean; + dcoDeclarationBypassed?: boolean; unverifiedCommits?: Array<{ sha: string; reason: string }>; }; }; @@ -298,20 +304,921 @@ function checkContributorApprovalOverlap( // Gate 1: CI green // --------------------------------------------------------------------------- +interface ExactDiffIdentity { + number: number; + headSha: string; + baseSha: string; + headRefName: string; + headRepository: string; +} + +interface E2eCoordinationEvidence { + valid: boolean | null; + startedAt?: number; + completedAt?: number; + trustedLegacyCheckId?: number; +} + +function parseGitHubTimestamp(value: string | undefined): number { + const match = value?.match(/^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.\d{1,3})?Z$/u); + if (!match) return Number.NaN; + const [year, month, day, hour, minute, second] = match.slice(1).map(Number); + const timestamp = Date.UTC(year, month - 1, day, hour, minute, second); + const parsed = new Date(timestamp); + return parsed.getUTCFullYear() === year && + parsed.getUTCMonth() === month - 1 && + parsed.getUTCDate() === day && + parsed.getUTCHours() === hour && + parsed.getUTCMinutes() === minute && + parsed.getUTCSeconds() === second + ? Date.parse(match[0]) + : Number.NaN; +} + +function fetchE2eCoordinationEvidence( + repo: string, + exactDiff: ExactDiffIdentity, +): E2eCoordinationEvidence { + const checkNames = ["E2E / PR Gate Coordination", "E2E / PR Gate"]; + const checkRuns: Array> = []; + const ids = new Set(); + for (const checkName of checkNames) { + const pages = ghJson([ + "api", + "--paginate", + "--slurp", + `repos/${repo}/commits/${exactDiff.headSha}/check-runs?check_name=${encodeURIComponent(checkName)}&filter=all&per_page=100`, + ]); + if (!Array.isArray(pages) || pages.length === 0) return { valid: null }; + + let expectedTotal: number | null = null; + let observedTotal = 0; + for (const page of pages) { + if (typeof page !== "object" || page === null || Array.isArray(page)) { + return { valid: null }; + } + const { total_count: totalCount, check_runs: pageRuns } = page as Record; + if ( + !Number.isSafeInteger(totalCount) || + (totalCount as number) < 0 || + (expectedTotal !== null && totalCount !== expectedTotal) || + !Array.isArray(pageRuns) + ) { + return { valid: null }; + } + expectedTotal = totalCount as number; + observedTotal += pageRuns.length; + for (const value of pageRuns) { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + return { valid: null }; + } + const record = value as Record; + if ( + !Number.isSafeInteger(record.id) || + (record.id as number) < 1 || + ids.has(record.id as number) || + (typeof record.external_id !== "string" && record.external_id !== null) + ) { + return { valid: null }; + } + ids.add(record.id as number); + checkRuns.push(record); + } + } + if (expectedTotal === null || observedTotal !== expectedTotal) { + return { valid: null }; + } + } + + const externalId = `nemoclaw-pr-e2e:v2:${exactDiff.number}:${exactDiff.headSha}:${exactDiff.baseSha}`; + const exactChecks = checkRuns.filter((check) => check.external_id === externalId); + if (exactChecks.length !== 1) return { valid: false }; + const exact = exactChecks[0]; + const app = exact.app; + const startedAt = + typeof exact.started_at === "string" ? parseGitHubTimestamp(exact.started_at) : Number.NaN; + const completedAt = + typeof exact.completed_at === "string" ? parseGitHubTimestamp(exact.completed_at) : Number.NaN; + const valid = + typeof exact.name === "string" && + checkNames.includes(exact.name) && + exact.head_sha === exactDiff.headSha && + typeof app === "object" && + app !== null && + !Array.isArray(app) && + (app as Record).id === 15368 && + exact.status === "completed" && + exact.conclusion === "success" && + Number.isFinite(startedAt) && + Number.isFinite(completedAt) && + startedAt <= completedAt; + return { + valid, + ...(valid ? { startedAt, completedAt } : {}), + ...(valid && exact.name === "E2E / PR Gate" + ? { trustedLegacyCheckId: exact.id as number } + : {}), + }; +} + +const ACTION_STATUSES = new Set([ + "COMPLETED", + "IN_PROGRESS", + "PENDING", + "QUEUED", + "REQUESTED", + "WAITING", +]); +const ACTION_CONCLUSIONS = new Set([ + "ACTION_REQUIRED", + "CANCELLED", + "FAILURE", + "NEUTRAL", + "SKIPPED", + "STALE", + "STARTUP_FAILURE", + "SUCCESS", + "TIMED_OUT", +]); +const PASSING_ACTION_RUN_CONCLUSIONS = new Set(["NEUTRAL", "SKIPPED", "SUCCESS"]); +const HEAD_BOUND_ACTION_EVENTS = new Set(["dynamic", "push", "workflow_call", "workflow_dispatch"]); +const PR_CI_RUN_TITLE = + /^CI PR #([1-9][0-9]*) head ([a-f0-9]{40}) base ([a-f0-9]{40}) gate (true|false)$/u; +const INSTALLER_HASH_RUN_TITLE = + /^Installer Hash PR #([1-9][0-9]*) head ([a-f0-9]{40}) base ([a-f0-9]{40}) gate (true|false)$/u; +const E2E_GATE_RUN_TITLE = + /^E2E Gate PR #([1-9][0-9]*) head ([a-f0-9]{40}) base ([a-f0-9]{40}) gate (true|false)$/u; +const REQUIRED_CHECK_WORKFLOW_PATHS = new Map([ + ["checks", ".github/workflows/pr.yaml"], + ["changes", ".github/workflows/pr.yaml"], + ["check-hash", ".github/workflows/installer-hash-check.yaml"], + ["commit-lint", ".github/workflows/commit-lint.yaml"], + ["dco-check", ".github/workflows/dco-check.yaml"], + ["E2E / PR Gate", ".github/workflows/pr-e2e-gate.yaml"], +]); +const PR_METADATA_EDIT_JOB_NAMES = new Set([ + "build-typecheck", + "changes", + "checks", + "cli-test-shards", + "cli-tests", + "docs-only-checks", + "installer-integration", + "plugin-tests", + "reviewed-npm-audit", + "static-checks", + "wechat-runtime-audit", +]); + +interface ActionRunMetadata { + attempt: number; + createdAt: number; + updatedAt: number; + exactDiff: boolean | null; + hasPullRequests: boolean | null; + headShaMatches: boolean | null; + headRefNameMatches: boolean | null; + headRepositoryMatches: boolean | null; + immutablePrDiff: boolean | null; + prCiGate: boolean | null; + installerHashGate: boolean | null; + e2eGateDiff: boolean | null; + e2eGateRun: boolean | null; + event: string | null; + path: string | null; + status: string | null; + conclusion: string | null; +} + +interface ActionJobMetadata { + name: string; + status: string; + conclusion: string | null; +} + +interface CurrentCheckRollup { + checks: StatusCheck[]; + incompleteAttemptEvidence: string[]; +} + +function currentCheckRollup( + statusCheckRollup: StatusCheck[], + repo: string, + exactDiff: ExactDiffIdentity, + e2eCoordinationEvidence: E2eCoordinationEvidence, +): CurrentCheckRollup { + const actionRunMetadataById = new Map(); + const latestAttemptJobsByRun = new Map | null>(); + const incompleteAttemptEvidence = new Set(); + + const fetchActionRunMetadata = (runId: string): ActionRunMetadata | null => { + const runData = ghJson(["api", `repos/${repo}/actions/runs/${runId}`]); + if (typeof runData !== "object" || runData === null || Array.isArray(runData)) { + return null; + } + const record = runData as Record; + if (!Number.isSafeInteger(record.run_attempt) || (record.run_attempt as number) < 1) { + return null; + } + const status = typeof record.status === "string" ? record.status.toUpperCase() : null; + const conclusion = + typeof record.conclusion === "string" ? record.conclusion.toUpperCase() : record.conclusion; + if ( + !status || + !ACTION_STATUSES.has(status) || + (conclusion !== null && + (typeof conclusion !== "string" || !ACTION_CONCLUSIONS.has(conclusion))) + ) { + return null; + } + + let exactDiffMatch: boolean | null = null; + let hasPullRequests: boolean | null = null; + if (Array.isArray(record.pull_requests)) { + hasPullRequests = record.pull_requests.length > 0; + exactDiffMatch = hasPullRequests ? false : null; + for (const value of record.pull_requests) { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + exactDiffMatch = null; + hasPullRequests = null; + break; + } + const pull = value as Record; + const head = pull.head; + const base = pull.base; + if ( + !Number.isSafeInteger(pull.number) || + typeof head !== "object" || + head === null || + Array.isArray(head) || + typeof base !== "object" || + base === null || + Array.isArray(base) || + typeof (head as Record).sha !== "string" || + typeof (base as Record).sha !== "string" + ) { + exactDiffMatch = null; + hasPullRequests = null; + break; + } + if ( + pull.number === exactDiff.number && + (head as Record).sha === exactDiff.headSha && + (base as Record).sha === exactDiff.baseSha + ) { + exactDiffMatch = true; + } + } + } + + const event = typeof record.event === "string" ? record.event : null; + const path = typeof record.path === "string" ? record.path : null; + const createdAt = + typeof record.created_at === "string" ? parseGitHubTimestamp(record.created_at) : Number.NaN; + const updatedAt = + typeof record.updated_at === "string" ? parseGitHubTimestamp(record.updated_at) : Number.NaN; + if (!Number.isFinite(createdAt) || !Number.isFinite(updatedAt) || createdAt > updatedAt) { + return null; + } + let immutablePrDiff: boolean | null = null; + let prCiGate: boolean | null = null; + let installerHashGate: boolean | null = null; + let e2eGateDiff: boolean | null = null; + let e2eGateRun: boolean | null = null; + if (event === "pull_request") { + const title = typeof record.display_title === "string" ? record.display_title : ""; + const titlePattern = + path === ".github/workflows/pr.yaml" + ? PR_CI_RUN_TITLE + : path === ".github/workflows/installer-hash-check.yaml" + ? INSTALLER_HASH_RUN_TITLE + : null; + const match = titlePattern ? title.match(titlePattern) : null; + if (match) { + const titlePrNumber = Number(match[1]); + if (Number.isSafeInteger(titlePrNumber) && titlePrNumber > 0) { + immutablePrDiff = + titlePrNumber === exactDiff.number && + match[2] === exactDiff.headSha && + match[3] === exactDiff.baseSha; + if (path === ".github/workflows/pr.yaml") { + prCiGate = match[4] === "true"; + } else if (path === ".github/workflows/installer-hash-check.yaml") { + installerHashGate = match[4] === "true"; + } + } + } + } + if (event === "pull_request_target" && path === ".github/workflows/pr-e2e-gate.yaml") { + const title = typeof record.display_title === "string" ? record.display_title : ""; + const match = title.match(E2E_GATE_RUN_TITLE); + if (match) { + const titlePrNumber = Number(match[1]); + if (Number.isSafeInteger(titlePrNumber) && titlePrNumber > 0) { + e2eGateDiff = + titlePrNumber === exactDiff.number && + match[2] === exactDiff.headSha && + match[3] === exactDiff.baseSha; + e2eGateRun = match[4] === "true"; + } + } + } + + const headRepository = record.head_repository; + + return { + attempt: record.run_attempt as number, + createdAt, + updatedAt, + exactDiff: exactDiffMatch, + hasPullRequests, + headShaMatches: + typeof record.head_sha === "string" ? record.head_sha === exactDiff.headSha : null, + headRefNameMatches: + typeof record.head_branch === "string" + ? record.head_branch === exactDiff.headRefName + : null, + headRepositoryMatches: + typeof headRepository === "object" && + headRepository !== null && + !Array.isArray(headRepository) && + typeof (headRepository as Record).full_name === "string" + ? (headRepository as Record).full_name === exactDiff.headRepository + : null, + immutablePrDiff, + prCiGate, + installerHashGate, + e2eGateDiff, + e2eGateRun, + event, + path, + status, + conclusion, + }; + }; + + const actionRunMetadata = (runId: string): ActionRunMetadata | null => { + if (actionRunMetadataById.has(runId)) return actionRunMetadataById.get(runId) ?? null; + const metadata = fetchActionRunMetadata(runId); + actionRunMetadataById.set(runId, metadata); + return metadata; + }; + + const latestAttemptJobs = (runId: string): Map | null => { + if (latestAttemptJobsByRun.has(runId)) return latestAttemptJobsByRun.get(runId) ?? null; + + const metadata = actionRunMetadata(runId); + if (!metadata) { + latestAttemptJobsByRun.set(runId, null); + return null; + } + const pages = ghJson([ + "api", + "--paginate", + "--slurp", + `repos/${repo}/actions/runs/${runId}/attempts/${metadata.attempt}/jobs?per_page=100`, + ]); + if (!Array.isArray(pages) || pages.length === 0) { + latestAttemptJobsByRun.set(runId, null); + return null; + } + + let expectedTotal: number | null = null; + const jobsById = new Map(); + let observedJobs = 0; + for (const page of pages) { + if (typeof page !== "object" || page === null || Array.isArray(page)) { + latestAttemptJobsByRun.set(runId, null); + return null; + } + const { jobs, total_count: totalCount } = page as Record; + if ( + !Number.isSafeInteger(totalCount) || + (totalCount as number) < 0 || + (expectedTotal !== null && totalCount !== expectedTotal) || + !Array.isArray(jobs) + ) { + latestAttemptJobsByRun.set(runId, null); + return null; + } + expectedTotal = totalCount as number; + observedJobs += jobs.length; + for (const value of jobs) { + if (typeof value !== "object" || value === null || Array.isArray(value)) { + latestAttemptJobsByRun.set(runId, null); + return null; + } + const { id, name, status, conclusion } = value as Record; + const normalizedStatus = typeof status === "string" ? status.toUpperCase() : null; + const normalizedConclusion = + typeof conclusion === "string" ? conclusion.toUpperCase() : conclusion; + if ( + !Number.isSafeInteger(id) || + (id as number) < 1 || + typeof name !== "string" || + !name || + !normalizedStatus || + !ACTION_STATUSES.has(normalizedStatus) || + (normalizedConclusion !== null && + (typeof normalizedConclusion !== "string" || + !ACTION_CONCLUSIONS.has(normalizedConclusion))) + ) { + latestAttemptJobsByRun.set(runId, null); + return null; + } + jobsById.set(String(id), { + name, + status: normalizedStatus, + conclusion: normalizedConclusion, + }); + } + } + if ( + expectedTotal === null || + observedJobs !== expectedTotal || + jobsById.size !== expectedTotal + ) { + latestAttemptJobsByRun.set(runId, null); + return null; + } + const refreshed = fetchActionRunMetadata(runId); + if ( + !refreshed || + refreshed.attempt !== metadata.attempt || + refreshed.createdAt !== metadata.createdAt || + refreshed.updatedAt !== metadata.updatedAt || + refreshed.exactDiff !== metadata.exactDiff || + refreshed.hasPullRequests !== metadata.hasPullRequests || + refreshed.headShaMatches !== metadata.headShaMatches || + refreshed.headRefNameMatches !== metadata.headRefNameMatches || + refreshed.headRepositoryMatches !== metadata.headRepositoryMatches || + refreshed.immutablePrDiff !== metadata.immutablePrDiff || + refreshed.prCiGate !== metadata.prCiGate || + refreshed.installerHashGate !== metadata.installerHashGate || + refreshed.e2eGateDiff !== metadata.e2eGateDiff || + refreshed.e2eGateRun !== metadata.e2eGateRun || + refreshed.event !== metadata.event || + refreshed.path !== metadata.path || + refreshed.status !== metadata.status || + refreshed.conclusion !== metadata.conclusion + ) { + latestAttemptJobsByRun.set(runId, null); + return null; + } + actionRunMetadataById.set(runId, refreshed); + latestAttemptJobsByRun.set(runId, jobsById); + return jobsById; + }; + + const classifyPrMetadataEditRun = ( + runId: string, + ): "recognized" | "invalid" | "not_metadata_edit" => { + const run = actionRunMetadata(runId); + const jobs = latestAttemptJobs(runId); + if (!run || !jobs || jobs.size === 0) return "not_metadata_edit"; + + if ( + runIdentityEvidence(runId, true) !== "current" || + run.event !== "pull_request" || + run.path !== ".github/workflows/pr.yaml" || + run.status !== "COMPLETED" || + run.conclusion !== "SUCCESS" + ) { + return "not_metadata_edit"; + } + if (run.prCiGate !== false) return "not_metadata_edit"; + + const jobNames = new Set([...jobs.values()].map((job) => job.name)); + const hasExactMetadataEditShape = + jobs.size === PR_METADATA_EDIT_JOB_NAMES.size && + jobNames.size === PR_METADATA_EDIT_JOB_NAMES.size && + [...PR_METADATA_EDIT_JOB_NAMES].every((name) => jobNames.has(name)) && + [...jobs.values()].every( + (job) => + job.status === "COMPLETED" && + (job.name === "checks" ? job.conclusion === "SUCCESS" : job.conclusion === "SKIPPED"), + ); + return hasExactMetadataEditShape ? "recognized" : "invalid"; + }; + + const e2eControllerHeadBinding = (run: ActionRunMetadata): "current" | "other" | "unknown" => { + if (run.event !== "pull_request_target" || run.path !== ".github/workflows/pr-e2e-gate.yaml") { + return "unknown"; + } + if ( + run.exactDiff === false || + run.e2eGateDiff === false || + run.headShaMatches === false || + run.headRefNameMatches === false || + run.headRepositoryMatches === false + ) { + return "other"; + } + return e2eCoordinationEvidence.valid === true && + run.e2eGateDiff === true && + run.hasPullRequests === false && + run.headShaMatches === true && + run.headRefNameMatches === true && + run.headRepositoryMatches === true + ? "current" + : "unknown"; + }; + + const e2eCoordinationIsEnclosed = (run: ActionRunMetadata): boolean => + e2eControllerHeadBinding(run) === "current" && + run.e2eGateRun === true && + e2eCoordinationEvidence.startedAt !== undefined && + e2eCoordinationEvidence.completedAt !== undefined && + run.createdAt <= e2eCoordinationEvidence.startedAt && + e2eCoordinationEvidence.completedAt <= run.updatedAt; + + const isNonAttemptRun = (runId: string): boolean => { + const run = actionRunMetadata(runId); + const jobs = latestAttemptJobs(runId); + if (!run || !jobs || jobs.size === 0) return false; + + const allSkippedTargetRun = Boolean( + (runIdentityEvidence(runId, true) === "current" || + e2eControllerHeadBinding(run) === "current") && + run.event === "pull_request_target" && + run.path === ".github/workflows/pr-e2e-gate.yaml" && + run.e2eGateDiff === true && + run.e2eGateRun === false && + run.status === "COMPLETED" && + run.conclusion === "SKIPPED" && + [...jobs.values()].every( + (job) => job.status === "COMPLETED" && job.conclusion === "SKIPPED", + ), + ); + if (allSkippedTargetRun) return true; + return classifyPrMetadataEditRun(runId) === "recognized"; + }; + + const isMeaningfulExactDiffRun = (runId: string, event: string, path: string): boolean => { + const run = actionRunMetadata(runId); + const jobs = latestAttemptJobs(runId); + return Boolean( + run && + jobs && + classifyPrMetadataEditRun(runId) === "not_metadata_edit" && + runIdentityEvidence(runId, true) === "current" && + run.event === event && + run.path === path && + (run.path !== ".github/workflows/pr.yaml" || run.prCiGate === true) && + (run.path !== ".github/workflows/installer-hash-check.yaml" || + run.installerHashGate === true) && + (run.path !== ".github/workflows/pr-e2e-gate.yaml" || + run.event !== "pull_request_target" || + (run.e2eGateDiff === true && run.e2eGateRun === true)) && + run.status === "COMPLETED" && + run.conclusion !== null && + run.conclusion !== "SKIPPED" && + jobs.size > 0 && + [...jobs.values()].every((job) => job.status === "COMPLETED" && job.conclusion !== null) && + [...jobs.values()].some((job) => job.conclusion !== "SKIPPED"), + ); + }; + + const checksFromLatestAttempt = (runId: string, checks: StatusCheck[]): StatusCheck[] | null => { + const checkName = checks[0]?.name; + const jobsById = latestAttemptJobs(runId); + if (!checkName || !jobsById) return null; + + const expectedIds = new Set( + [...jobsById].filter(([, job]) => job.name === checkName).map(([id]) => id), + ); + if (expectedIds.size === 0) return null; + + const selected: StatusCheck[] = []; + const selectedIds = new Set(); + for (const check of checks) { + const match = check.detailsUrl?.match( + new RegExp(`/actions/runs/${runId}/job/(\\d+)(?:[/?#]|$)`, "u"), + ); + if (!match) return null; + if (expectedIds.has(match[1])) { + if (selectedIds.has(match[1])) return null; + selectedIds.add(match[1]); + selected.push(check); + } + } + return selectedIds.size === expectedIds.size ? selected : null; + }; + + const latestAttemptChecks = (runId: string, checks: StatusCheck[]): StatusCheck[] => { + const selected = checksFromLatestAttempt(runId, checks); + const checkName = checks[0]?.name ?? ""; + const requiresExactDiff = REQUIRED_CHECK_NAMES.includes(checkName); + const expectedWorkflowPath = REQUIRED_CHECK_WORKFLOW_PATHS.get(checkName); + const runMetadata = actionRunMetadata(runId); + const hasCurrentIdentity = runIdentityEvidence(runId, requiresExactDiff) === "current"; + if ( + !selected || + !hasCurrentIdentity || + !runMetadata || + !runMetadata.event || + !runMetadata.path || + (expectedWorkflowPath !== undefined && runMetadata.path !== expectedWorkflowPath) || + (expectedWorkflowPath === ".github/workflows/pr.yaml" && runMetadata.prCiGate !== true) || + (expectedWorkflowPath === ".github/workflows/installer-hash-check.yaml" && + runMetadata.installerHashGate !== true) || + (expectedWorkflowPath === ".github/workflows/pr-e2e-gate.yaml" && + runMetadata.event === "pull_request_target" && + (runMetadata.e2eGateDiff !== true || runMetadata.e2eGateRun !== true)) || + runMetadata.status !== "COMPLETED" || + runMetadata.conclusion === null || + (requiresExactDiff + ? runMetadata.conclusion !== "SUCCESS" + : !PASSING_ACTION_RUN_CONCLUSIONS.has(runMetadata.conclusion)) + ) { + incompleteAttemptEvidence.add(checks[0]?.name ?? "(unknown)"); + } + return selected ?? checks; + }; + + const actionRunId = (check: StatusCheck): string | undefined => + check.detailsUrl?.match(/\/actions\/runs\/(\d+)(?:\/|$)/)?.[1]; + + const isTrustedLegacyE2eCheck = (check: StatusCheck): boolean => + e2eCoordinationEvidence.trustedLegacyCheckId !== undefined && + check.name === "E2E / PR Gate" && + check.detailsUrl?.match(/\/runs\/(\d+)(?:[/?#]|$)/u)?.[1] === + String(e2eCoordinationEvidence.trustedLegacyCheckId); + + function runIdentityEvidence( + runId: string, + requiresExactDiff: boolean, + ): "current" | "other" | "unknown" { + const metadata = actionRunMetadata(runId); + if (!metadata?.event || !metadata.path) return "unknown"; + if ( + metadata.event === "pull_request" && + metadata.path === ".github/workflows/installer-hash-check.yaml" + ) { + if ( + metadata.immutablePrDiff === false || + metadata.exactDiff === false || + metadata.headShaMatches === false + ) { + return "other"; + } + if ( + metadata.immutablePrDiff === true && + metadata.headShaMatches === true && + (metadata.exactDiff === true || metadata.hasPullRequests === false) + ) { + return "current"; + } + return "unknown"; + } + if (metadata.exactDiff === true) { + if (metadata.headShaMatches === true) { + if (metadata.event === "pull_request" && metadata.path === ".github/workflows/pr.yaml") { + if (metadata.immutablePrDiff === true) return "current"; + if (metadata.immutablePrDiff === false) return "other"; + return "unknown"; + } + return "current"; + } + if (metadata.headShaMatches === false) return "other"; + return "unknown"; + } + if (metadata.exactDiff === false) return "other"; + const e2eHeadBinding = e2eControllerHeadBinding(metadata); + if (e2eHeadBinding === "other") return "other"; + if (e2eHeadBinding === "current") { + return e2eCoordinationIsEnclosed(metadata) ? "current" : "unknown"; + } + if ( + !requiresExactDiff && + metadata.hasPullRequests === false && + HEAD_BOUND_ACTION_EVENTS.has(metadata.event) && + metadata.headShaMatches !== null + ) { + return metadata.headShaMatches ? "current" : "other"; + } + return "unknown"; + } + + const allActionRunIds = new Set( + statusCheckRollup.map(actionRunId).filter((runId): runId is string => Boolean(runId)), + ); + const hasMeaningfulAlternateRun = (runId: string): boolean => { + const { event, path } = actionRunMetadata(runId) ?? {}; + return Boolean( + event && + path && + [...allActionRunIds].some( + (otherRunId) => otherRunId !== runId && isMeaningfulExactDiffRun(otherRunId, event, path), + ), + ); + }; + + const groups = new Map(); + for (const check of statusCheckRollup) { + const identity = JSON.stringify([ + check.__typename ?? (check.context ? "StatusContext" : "CheckRun"), + check.name ?? check.context ?? "(unknown)", + check.workflowName ?? "", + ]); + const group = groups.get(identity) ?? []; + group.push(check); + groups.set(identity, group); + } + + const current: StatusCheck[] = []; + for (const group of groups.values()) { + const groupName = group[0].name ?? group[0].context ?? "(unknown)"; + const requiredCheck = REQUIRED_CHECK_NAMES.includes(groupName); + const expectsActionEvidence = group.some( + (check) => + check.__typename !== "StatusContext" && + (check.detailsUrl?.includes("/actions/") || + (Boolean(check.workflowName) && !/\/runs\/\d+(?:[/?#]|$)/u.test(check.detailsUrl ?? ""))), + ); + if ( + (requiredCheck || expectsActionEvidence) && + group.some((check) => !actionRunId(check) && !isTrustedLegacyE2eCheck(check)) + ) { + incompleteAttemptEvidence.add(groupName); + } + if (group.length === 1) { + const runId = group[0].__typename !== "StatusContext" ? actionRunId(group[0]) : undefined; + if (runId && classifyPrMetadataEditRun(runId) === "invalid") { + incompleteAttemptEvidence.add(groupName); + } + if (runId && isNonAttemptRun(runId)) { + if (hasMeaningfulAlternateRun(runId)) continue; + incompleteAttemptEvidence.add(groupName); + } + current.push(...(runId ? latestAttemptChecks(runId, group) : group)); + continue; + } + + if (group[0].__typename !== "StatusContext") { + const hasCheckTimestampEvidence = group.every((check) => + Number.isFinite(parseGitHubTimestamp(check.startedAt ?? check.completedAt)), + ); + if (!hasCheckTimestampEvidence) { + incompleteAttemptEvidence.add(group[0]?.name ?? "(unknown)"); + } + const byRun = new Map(); + for (const check of group) { + const runId = actionRunId(check); + if (!runId) { + byRun.clear(); + break; + } + const runChecks = byRun.get(runId) ?? []; + runChecks.push(check); + byRun.set(runId, runChecks); + } + if (byRun.size > 1) { + const runs = [...byRun].map(([runId, checks]) => { + return { + runId, + checks, + timestamp: actionRunMetadata(runId)?.createdAt ?? Number.NaN, + }; + }); + const hasOrderingEvidence = + hasCheckTimestampEvidence && runs.every(({ timestamp }) => Number.isFinite(timestamp)); + if (!hasOrderingEvidence) { + incompleteAttemptEvidence.add(group[0]?.name ?? "(unknown)"); + } + if (hasOrderingEvidence) { + const currentIdentityRuns = runs.filter( + ({ runId }) => runIdentityEvidence(runId, requiredCheck) === "current", + ); + const unknownIdentityRun = runs.some( + ({ runId }) => + runIdentityEvidence(runId, requiredCheck) === "unknown" && !isNonAttemptRun(runId), + ); + const currentWorkflowIdentities = new Set( + currentIdentityRuns.map(({ runId }) => { + const metadata = actionRunMetadata(runId); + return metadata?.event && metadata.path + ? JSON.stringify([metadata.event, metadata.path]) + : null; + }), + ); + if ( + currentIdentityRuns.length === 0 || + unknownIdentityRun || + currentWorkflowIdentities.size !== 1 || + currentWorkflowIdentities.has(null) + ) { + incompleteAttemptEvidence.add(group[0]?.name ?? "(unknown)"); + } + const identityCandidates = currentIdentityRuns.length > 0 ? currentIdentityRuns : runs; + const candidates = identityCandidates.filter(({ runId }) => { + if (classifyPrMetadataEditRun(runId) === "invalid") { + incompleteAttemptEvidence.add(group[0]?.name ?? "(unknown)"); + return true; + } + if (!isNonAttemptRun(runId)) return true; + const hasMeaningfulRun = hasMeaningfulAlternateRun(runId); + if (!hasMeaningfulRun) { + incompleteAttemptEvidence.add(group[0]?.name ?? "(unknown)"); + } + return !hasMeaningfulRun; + }); + if (candidates.length === 0) continue; + const latestTimestamp = Math.max(...candidates.map(({ timestamp }) => timestamp)); + const latestRuns = candidates.filter(({ timestamp }) => timestamp === latestTimestamp); + for (const latest of latestRuns) { + current.push(...latestAttemptChecks(latest.runId, latest.checks)); + } + continue; + } + } + + if (byRun.size === 1) { + const [runId, checks] = [...byRun][0]; + current.push(...latestAttemptChecks(runId, checks)); + continue; + } + + const customCheckRuns = group.every( + (check) => + !check.detailsUrl?.includes("/actions/runs/") && + /\/runs\/\d+(?:[/?#]|$)/u.test(check.detailsUrl ?? ""), + ); + if (customCheckRuns) { + const timestamped = group.map((check) => ({ + check, + timestamp: parseGitHubTimestamp(check.startedAt ?? check.completedAt), + })); + if (timestamped.every(({ timestamp }) => Number.isFinite(timestamp))) { + const latestTimestamp = Math.max(...timestamped.map(({ timestamp }) => timestamp)); + current.push( + ...timestamped + .filter(({ timestamp }) => timestamp === latestTimestamp) + .map(({ check }) => check), + ); + continue; + } + } + + // Keep duplicate jobs from one workflow run together. This prevents a + // later-starting matrix job from hiding another job's failure. + current.push(...group); + continue; + } + + const timestamped = group.map((check) => ({ + check, + timestamp: parseGitHubTimestamp(check.startedAt ?? check.completedAt), + })); + if (timestamped.some(({ timestamp }) => !Number.isFinite(timestamp))) { + current.push(...group); + continue; + } + const latestTimestamp = Math.max(...timestamped.map(({ timestamp }) => timestamp)); + current.push( + ...timestamped + .filter(({ timestamp }) => timestamp === latestTimestamp) + .map(({ check }) => check), + ); + } + const prCiRunIds = new Set(); + const prCiNames = new Set(); + for (const check of current) { + const name = check.name ?? check.context; + if (name !== "checks" && name !== "changes") continue; + prCiNames.add(name); + const runId = actionRunId(check); + if (runId) prCiRunIds.add(runId); + } + if (prCiNames.size === 2 && prCiRunIds.size !== 1) { + incompleteAttemptEvidence.add("checks"); + incompleteAttemptEvidence.add("changes"); + } + return { checks: current, incompleteAttemptEvidence: [...incompleteAttemptEvidence].sort() }; +} + function checkCi( statusCheckRollup: StatusCheck[] | null, + repo: string, + exactDiff: ExactDiffIdentity, ): GateResult & { failingChecks?: string[]; pendingChecks?: string[]; missingChecks?: string[] } { if (!statusCheckRollup || statusCheckRollup.length === 0) { return { pass: false, details: "No status checks found" }; } + const e2eCoordinationEvidence = fetchE2eCoordinationEvidence(repo, exactDiff); + const rollup = currentCheckRollup(statusCheckRollup, repo, exactDiff, e2eCoordinationEvidence); + const currentChecks = rollup.checks; + const incompleteAttemptEvidence = new Set(rollup.incompleteAttemptEvidence); + if (e2eCoordinationEvidence.valid !== true) { + incompleteAttemptEvidence.add("E2E / PR Gate"); + } + // Check that all required checks are present. // Fork PRs from first-time contributors need "Approve and run" before // pull_request workflows execute. Until then only pull_request_target // checks (like check-pr-limit) and external bots (CodeRabbit) appear. - const presentNames = new Set( - statusCheckRollup.map((c) => c.name ?? c.context ?? "").filter(Boolean), - ); + const presentNames = new Set(currentChecks.map((c) => c.name ?? c.context ?? "").filter(Boolean)); const missingChecks = REQUIRED_CHECK_NAMES.filter((name) => !presentNames.has(name)); if (missingChecks.length > 0) { return { @@ -325,7 +1232,7 @@ function checkCi( const failing: string[] = []; const pending: string[] = []; - for (const check of statusCheckRollup) { + for (const check of currentChecks) { const checkName = check.name ?? check.context ?? "(unknown)"; // StatusContext (e.g. CodeRabbit) uses `state` instead of `status`/`conclusion`. @@ -342,12 +1249,10 @@ function checkCi( // CheckRun uses `status` and `conclusion`. const conclusion = (check.conclusion ?? "").toUpperCase(); const status = (check.status ?? "").toUpperCase(); + const requiredCheck = REQUIRED_CHECK_NAMES.includes(checkName); if (status !== "COMPLETED") { pending.push(checkName); - } else if ( - !passing.has(conclusion) || - (checkName === "E2E / PR Gate" && conclusion !== "SUCCESS") - ) { + } else if (!passing.has(conclusion) || (requiredCheck && conclusion !== "SUCCESS")) { failing.push(`${checkName}: ${conclusion}`); } } @@ -363,21 +1268,102 @@ function checkCi( if (pending.length > 0) { return { pass: false, details: `${pending.length} pending check(s)`, pendingChecks: pending }; } - return { pass: true, details: `All ${statusCheckRollup.length} checks green` }; + if (incompleteAttemptEvidence.size > 0) { + const incompleteNames = [...incompleteAttemptEvidence].sort(); + return { + pass: false, + details: `${incompleteNames.length} check context(s) have incomplete latest-attempt evidence`, + failingChecks: incompleteNames.map((name) => `${name}: latest attempt evidence incomplete`), + }; + } + return { pass: true, details: `All ${currentChecks.length} current checks green` }; } // --------------------------------------------------------------------------- // Gate 2: No conflicts // --------------------------------------------------------------------------- -function checkConflicts(mergeStateStatus: string): GateResult & { mergeStateStatus?: string } { - const clean = ["CLEAN", "HAS_HOOKS", "UNSTABLE"]; +function checkConflicts( + mergeable: string, + mergeStateStatus: string, + baseSha: string, + currentBaseSha: string | null, +): GateResult & { + mergeable?: string; + mergeStateStatus?: string; + baseSha?: string; + currentBaseSha?: string; +} { + const conflictStatus = (mergeable ?? "UNKNOWN").toUpperCase(); const status = (mergeStateStatus ?? "UNKNOWN").toUpperCase(); + const currentBaseStates = new Set(["BLOCKED", "CLEAN", "HAS_HOOKS", "UNSTABLE"]); - if (clean.includes(status)) { - return { pass: true, details: "No merge conflicts", mergeStateStatus: status }; + if (!currentBaseSha) { + return { + pass: false, + details: "Unable to verify the current base branch revision", + mergeable: conflictStatus, + mergeStateStatus: status, + baseSha, + }; + } + if (baseSha !== currentBaseSha) { + return { + pass: false, + details: "PR branch is behind its base branch; refresh it before approval", + mergeable: conflictStatus, + mergeStateStatus: status, + baseSha, + currentBaseSha, + }; + } + if (conflictStatus === "MERGEABLE" && currentBaseStates.has(status)) { + return { + pass: true, + details: "No merge conflicts", + mergeable: conflictStatus, + mergeStateStatus: status, + baseSha, + currentBaseSha, + }; } - return { pass: false, details: `Merge state: ${status}`, mergeStateStatus: status }; + return { + pass: false, + details: + status === "BEHIND" + ? "PR branch is behind its base branch; refresh it before approval" + : `Mergeability: ${conflictStatus}; merge state: ${status}`, + mergeable: conflictStatus, + mergeStateStatus: status, + baseSha, + currentBaseSha, + }; +} + +function fetchCurrentBaseSha(repo: string, number: number): string | null { + const [owner, name, extra] = repo.split("/"); + if (!owner || !name || extra) return null; + + const response = ghJson([ + "api", + "graphql", + "-F", + `owner=${owner}`, + "-F", + `name=${name}`, + "-F", + `number=${number}`, + "-f", + `query=query CurrentBaseRef($owner: String!, $name: String!, $number: Int!) { + repository(owner: $owner, name: $name) { + pullRequest(number: $number) { baseRef { target { oid } } } + } + }`, + ]) as { + data?: { repository?: { pullRequest?: { baseRef?: { target?: { oid?: unknown } } } } }; + } | null; + const oid = response?.data?.repository?.pullRequest?.baseRef?.target?.oid; + return typeof oid === "string" && /^[0-9a-f]{40}$/i.test(oid) ? oid : null; } // --------------------------------------------------------------------------- @@ -537,6 +1523,7 @@ function checkRiskyCodeTested( // --------------------------------------------------------------------------- const DCO_DECLARATION = /^Signed-off-by:\s+.+\s+<[^<>\s]+@[^<>\s]+>\s*$/mu; +const DCO_BODY_BYPASS_AUTHORS = new Set(["app/dependabot", "dependabot[bot]"]); interface CommitVerificationRecord { sha: string; @@ -569,11 +1556,15 @@ function checkContributorCompliance( repo: string, number: number, body: string, + authorLogin: string | null, ): GateResult & { dcoDeclarationPresent?: boolean; + dcoDeclarationBypassed?: boolean; unverifiedCommits?: Array<{ sha: string; reason: string }>; } { const dcoDeclarationPresent = DCO_DECLARATION.test(body ?? ""); + const dcoDeclarationBypassed = + typeof authorLogin === "string" && DCO_BODY_BYPASS_AUTHORS.has(authorLogin.toLowerCase()); const raw = run("gh", [ "api", `repos/${repo}/pulls/${number}/commits`, @@ -587,6 +1578,7 @@ function checkContributorCompliance( pass: false, details: "Could not verify PR commit signatures (API error — fail-closed)", dcoDeclarationPresent, + dcoDeclarationBypassed, }; } @@ -601,6 +1593,7 @@ function checkContributorCompliance( pass: false, details: "Could not parse PR commit signature data — fail-closed", dcoDeclarationPresent, + dcoDeclarationBypassed, }; } @@ -609,15 +1602,18 @@ function checkContributorCompliance( pass: false, details: "No PR commits returned while checking contributor compliance — fail-closed", dcoDeclarationPresent, + dcoDeclarationBypassed, }; } const unverifiedCommits = commits .filter((commit) => commit.verified !== true) .map(({ sha, reason }) => ({ sha, reason })); - if (!dcoDeclarationPresent || unverifiedCommits.length > 0) { + if ((!dcoDeclarationPresent && !dcoDeclarationBypassed) || unverifiedCommits.length > 0) { const failures = [ - ...(dcoDeclarationPresent ? [] : ["PR body lacks a valid Signed-off-by declaration"]), + ...(dcoDeclarationPresent || dcoDeclarationBypassed + ? [] + : ["PR body lacks a valid Signed-off-by declaration"]), ...(unverifiedCommits.length > 0 ? [`${unverifiedCommits.length} commit(s) are not GitHub Verified`] : []), @@ -626,14 +1622,16 @@ function checkContributorCompliance( pass: false, details: failures.join("; "), dcoDeclarationPresent, + dcoDeclarationBypassed, unverifiedCommits, }; } return { pass: true, - details: `DCO declaration present; all ${commits.length} commit(s) are GitHub Verified`, + details: `${dcoDeclarationBypassed ? `PR-body DCO declaration bypassed for ${authorLogin}` : "DCO declaration present"}; all ${commits.length} commit(s) are GitHub Verified`, dcoDeclarationPresent, + dcoDeclarationBypassed, unverifiedCommits: [], }; } @@ -642,6 +1640,122 @@ function checkContributorCompliance( // Main // --------------------------------------------------------------------------- +interface PrRevisionSnapshot { + title: string; + body: string; + state: string; + isDraft: boolean; + mergeable: string; + mergeStateStatus: string; + headRefOid: string; + baseRefOid: string; + headRefName: string; + baseRefName: string; + headRepository: string; +} + +function fetchPrRevisionSnapshot(repo: string, number: number): PrRevisionSnapshot | null { + const value = ghJson([ + "pr", + "view", + String(number), + "--repo", + repo, + "--json", + "title,body,state,isDraft,mergeable,mergeStateStatus,headRefOid,baseRefOid,headRefName,baseRefName,headRepository", + ]); + if (typeof value !== "object" || value === null || Array.isArray(value)) return null; + const record = value as Record; + const headRepository = record.headRepository; + if ( + typeof record.title !== "string" || + typeof record.body !== "string" || + typeof record.state !== "string" || + typeof record.isDraft !== "boolean" || + typeof record.mergeable !== "string" || + typeof record.mergeStateStatus !== "string" || + typeof record.headRefOid !== "string" || + typeof record.baseRefOid !== "string" || + typeof record.headRefName !== "string" || + typeof record.baseRefName !== "string" || + typeof headRepository !== "object" || + headRepository === null || + Array.isArray(headRepository) || + typeof (headRepository as Record).nameWithOwner !== "string" + ) { + return null; + } + return { + title: record.title, + body: record.body, + state: record.state, + isDraft: record.isDraft, + mergeable: record.mergeable, + mergeStateStatus: record.mergeStateStatus, + headRefOid: record.headRefOid, + baseRefOid: record.baseRefOid, + headRefName: record.headRefName, + baseRefName: record.baseRefName, + headRepository: (headRepository as Record).nameWithOwner as string, + }; +} + +function checkFinalRevision( + captured: PrRevisionSnapshot, + current: PrRevisionSnapshot | null, + currentBaseSha: string | null, +): ReturnType { + if (!current) { + return { + pass: false, + details: "Unable to re-read the PR revision after gate evaluation", + mergeable: captured.mergeable, + mergeStateStatus: captured.mergeStateStatus, + baseSha: captured.baseRefOid, + }; + } + if (current.state.toUpperCase() !== "OPEN" || current.isDraft) { + return { + pass: false, + details: current.isDraft + ? "PR became a draft during gate evaluation" + : "PR is no longer open", + mergeable: current.mergeable, + mergeStateStatus: current.mergeStateStatus, + baseSha: current.baseRefOid, + ...(currentBaseSha ? { currentBaseSha } : {}), + }; + } + const changed = + current.title !== captured.title || + current.body !== captured.body || + current.state !== captured.state || + current.isDraft !== captured.isDraft || + current.mergeable !== captured.mergeable || + current.mergeStateStatus !== captured.mergeStateStatus || + current.headRefOid !== captured.headRefOid || + current.baseRefOid !== captured.baseRefOid || + current.headRefName !== captured.headRefName || + current.baseRefName !== captured.baseRefName || + current.headRepository !== captured.headRepository; + if (changed) { + return { + pass: false, + details: "PR revision or merge state changed during gate evaluation; rerun the gate checker", + mergeable: current.mergeable, + mergeStateStatus: current.mergeStateStatus, + baseSha: current.baseRefOid, + ...(currentBaseSha ? { currentBaseSha } : {}), + }; + } + return checkConflicts( + current.mergeable, + current.mergeStateStatus, + current.baseRefOid, + currentBaseSha, + ); +} + function main(): void { const args = process.argv.slice(2); const prNumber = parseInt(args[0], 10); @@ -659,7 +1773,7 @@ function main(): void { "--repo", repo, "--json", - "number,title,url,body,files,statusCheckRollup,mergeStateStatus,author", + "number,title,url,body,files,statusCheckRollup,state,isDraft,mergeable,mergeStateStatus,headRefOid,baseRefOid,headRefName,baseRefName,headRepository,author", ]) as { number: number; title: string; @@ -667,7 +1781,15 @@ function main(): void { body: string; files: Array<{ path: string; status: string }>; statusCheckRollup: StatusCheck[]; + state: string; + isDraft: boolean; + mergeable: string; mergeStateStatus: string; + headRefOid: string; + baseRefOid: string; + headRefName: string; + baseRefName: string; + headRepository: { nameWithOwner: string }; author: PrIdentity | null; } | null; @@ -676,16 +1798,45 @@ function main(): void { process.exit(1); } - const ci = checkCi(prData.statusCheckRollup); - const conflicts = checkConflicts(prData.mergeStateStatus); + const ci = checkCi(prData.statusCheckRollup, repo, { + number: prNumber, + headSha: prData.headRefOid, + baseSha: prData.baseRefOid, + headRefName: prData.headRefName, + headRepository: prData.headRepository.nameWithOwner, + }); const coderabbit = checkCodeRabbit(repo, prNumber); const riskyCodeTested = checkRiskyCodeTested(prData.files ?? []); - const contributorCompliance = checkContributorCompliance(repo, prNumber, prData.body ?? ""); + const contributorCompliance = checkContributorCompliance( + repo, + prNumber, + prData.body ?? "", + prData.author?.login ?? null, + ); const contributorApprovalHistory = fetchContributorApprovalHistory(repo, prNumber); const contributorApprovalOverlap = checkContributorApprovalOverlap( prData, contributorApprovalHistory, ); + const currentBaseSha = fetchCurrentBaseSha(repo, prNumber); + const currentRevision = fetchPrRevisionSnapshot(repo, prNumber); + const conflicts = checkFinalRevision( + { + title: prData.title, + body: prData.body, + state: prData.state, + isDraft: prData.isDraft, + mergeable: prData.mergeable, + mergeStateStatus: prData.mergeStateStatus, + headRefOid: prData.headRefOid, + baseRefOid: prData.baseRefOid, + headRefName: prData.headRefName, + baseRefName: prData.baseRefName, + headRepository: prData.headRepository.nameWithOwner, + }, + currentRevision, + currentBaseSha, + ); const output: GateOutput = { pr: prNumber, diff --git a/.agents/skills/nemoclaw-maintainer-day/scripts/shared.ts b/.agents/skills/nemoclaw-maintainer-day/scripts/shared.ts index caa0cb6d3c..85e8794249 100644 --- a/.agents/skills/nemoclaw-maintainer-day/scripts/shared.ts +++ b/.agents/skills/nemoclaw-maintainer-day/scripts/shared.ts @@ -104,6 +104,10 @@ export interface StatusCheck { __typename?: string; name?: string; // CheckRun field context?: string; // StatusContext field + workflowName?: string; // CheckRun workflow identity + startedAt?: string; // CheckRun/StatusContext RFC3339 timestamp + completedAt?: string; // CheckRun RFC3339 timestamp + detailsUrl?: string; // CheckRun workflow-run identity when Actions produced it status?: string; // CheckRun: COMPLETED, IN_PROGRESS, QUEUED, etc. conclusion?: string; // CheckRun: SUCCESS, FAILURE, NEUTRAL, SKIPPED, etc. state?: string; // StatusContext: SUCCESS, FAILURE, PENDING, ERROR diff --git a/.agents/skills/nemoclaw-maintainer-pr-comparator/scripts/collect-gates.sh b/.agents/skills/nemoclaw-maintainer-pr-comparator/scripts/collect-gates.sh index 31130fccf1..7616893afa 100755 --- a/.agents/skills/nemoclaw-maintainer-pr-comparator/scripts/collect-gates.sh +++ b/.agents/skills/nemoclaw-maintainer-pr-comparator/scripts/collect-gates.sh @@ -41,7 +41,7 @@ else fi raw=$(gh pr view "$pr" "${repo_args[@]}" \ - --json number,state,body,headRefOid,statusCheckRollup,mergeable,mergeStateStatus,reviewDecision \ + --json number,state,body,author,headRefOid,statusCheckRollup,mergeable,mergeStateStatus,reviewDecision \ 2>/dev/null) || { emit_error "fetch_failed" exit 0 @@ -106,6 +106,12 @@ if printf '%s' "$raw" | jq -r '.body // ""' | grep -Eq '^Signed-off-by:[[:space: else dco_declaration_present=false fi +author_login=$(printf '%s' "$raw" | jq -r '.author.login // ""' | tr '[:upper:]' '[:lower:]') +if [ "$author_login" = "app/dependabot" ] || [ "$author_login" = "dependabot[bot]" ]; then + dco_declaration_bypassed=true +else + dco_declaration_bypassed=false +fi commits_fetch_failed=false commit_parse_failed=false @@ -122,7 +128,8 @@ elif commits_json=$(printf '%s\n' "$commits_raw" | jq -s '.' 2>/dev/null); then unverified_commits=$(printf '%s' "$commits_json" | jq '[.[] | select(.verified != true) | {sha, reason}]') unverified_count=$(printf '%s' "$unverified_commits" | jq 'length') gate_contributor_compliance=$( - [ "$dco_declaration_present" = "true" ] && [ "$unverified_count" = "0" ] && echo true || echo false + { [ "$dco_declaration_present" = "true" ] || [ "$dco_declaration_bypassed" = "true" ]; } \ + && [ "$unverified_count" = "0" ] && echo true || echo false ) else commit_count=0 @@ -166,6 +173,7 @@ jq -n \ --arg mergeable "$mergeable" \ --arg merge_state "$merge_state" \ --argjson dco_declaration_present "$dco_declaration_present" \ + --argjson dco_declaration_bypassed "$dco_declaration_bypassed" \ --argjson commit_count "$commit_count" \ --argjson unverified_commits "$unverified_commits" \ --argjson commit_fetch_failed "$commits_fetch_failed" \ @@ -192,6 +200,7 @@ jq -n \ mergeable: $mergeable, merge_state_status: $merge_state, dco_declaration_present: $dco_declaration_present, + dco_declaration_bypassed: $dco_declaration_bypassed, commit_count: $commit_count, unverified_commits: $unverified_commits, commit_fetch_failed: $commit_fetch_failed, diff --git a/.github/workflows/dco-check.yaml b/.github/workflows/dco-check.yaml index b0f3113c2f..7eebff250e 100644 --- a/.github/workflows/dco-check.yaml +++ b/.github/workflows/dco-check.yaml @@ -16,26 +16,24 @@ concurrency: jobs: dco-check: - if: github.event.pull_request.user.login != 'dependabot[bot]' runs-on: ubuntu-latest timeout-minutes: 5 permissions: contents: read steps: - - name: Check out repository - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 - - - name: Check DCO bypass list + - name: Check Dependabot DCO bypass id: dco-bypass env: USERNAME: ${{ github.event.pull_request.user.login }} run: | - if grep -Fxq "$USERNAME" .github/dco-bypass.txt; then + # GitHub exposes the same trusted app as dependabot[bot] in event + # payloads and app/dependabot through some GraphQL PR surfaces. + if [[ "$USERNAME" == "dependabot[bot]" || "$USERNAME" == "app/dependabot" ]]; then echo "bypass=true" >> "$GITHUB_OUTPUT" - echo "Author is in dco-bypass.txt; skipping DCO sign-off requirement." + echo "Dependabot is exempt from the PR-body DCO declaration." else echo "bypass=false" >> "$GITHUB_OUTPUT" - echo "Author is not in dco-bypass.txt; DCO sign-off is required." + echo "The PR-body DCO declaration is required." fi - name: Check PR body for Signed-off-by diff --git a/.github/workflows/installer-hash-check.yaml b/.github/workflows/installer-hash-check.yaml index ef8d02ba2b..2ea04e28cd 100644 --- a/.github/workflows/installer-hash-check.yaml +++ b/.github/workflows/installer-hash-check.yaml @@ -11,9 +11,16 @@ name: Security / Installer Hash Check +run-name: >- + ${{ github.event_name == 'pull_request' && + format('Installer Hash PR #{0} head {1} base {2} gate true', + github.event.pull_request.number, github.event.pull_request.head.sha, + github.event.pull_request.base.sha) || + format('Installer Hash {0} {1}', github.event_name, github.sha) }} + on: pull_request: - types: [opened, synchronize, reopened] + types: [opened, synchronize, reopened, edited] push: branches: [main] schedule: diff --git a/.github/workflows/pr-e2e-gate.yaml b/.github/workflows/pr-e2e-gate.yaml index b2e29e91ba..b17c6789ee 100644 --- a/.github/workflows/pr-e2e-gate.yaml +++ b/.github/workflows/pr-e2e-gate.yaml @@ -3,6 +3,14 @@ name: E2E / PR Gate Controller +run-name: >- + ${{ github.event_name == 'pull_request_target' && + format('E2E Gate PR #{0} head {1} base {2} gate {3}', + github.event.pull_request.number, github.event.pull_request.head.sha, + github.event.pull_request.base.sha, + github.event.action != 'edited' || github.event.changes.base != null) || + format('E2E Gate {0} {1}', github.event_name, github.run_id) }} + on: workflow_run: workflows: ["CI / Pull Request"] diff --git a/ci/source-shape-test-budget.json b/ci/source-shape-test-budget.json index 813ef6a288..d444e1e10a 100644 --- a/ci/source-shape-test-budget.json +++ b/ci/source-shape-test-budget.json @@ -396,6 +396,16 @@ "test": "publishes coverage only from same-repository code (#6692)", "category": "security" }, + { + "file": "test/pr-workflow-contract.test.ts", + "test": "records the Dependabot DCO bypass as a successful required job", + "category": "security" + }, + { + "file": "test/pr-workflow-contract.test.ts", + "test": "reruns installer hash verification after a pull request base retarget", + "category": "security" + }, { "file": "test/pr-workflow-contract.test.ts", "test": "reuses the same shared CI actions in PR and main workflows", diff --git a/test/pr-e2e-gate-workflow.test.ts b/test/pr-e2e-gate-workflow.test.ts index e53db22e56..6b7365578f 100644 --- a/test/pr-e2e-gate-workflow.test.ts +++ b/test/pr-e2e-gate-workflow.test.ts @@ -26,6 +26,7 @@ type CoordinatorJob = WorkflowJob & { type TriggeredWorkflow = Omit & { name: string; + "run-name": string; on: { workflow_run: { workflows: string[]; types: string[] }; pull_request_target: { types: string[] }; @@ -304,6 +305,11 @@ describe("PR E2E gate workflow", () => { const recordForkSkip = workflow.jobs["record-fork-e2e-skip"]; expect(workflow.name).toBe("E2E / PR Gate Controller"); + expect(workflow["run-name"]).toContain("E2E Gate PR #{0} head {1} base {2} gate {3}"); + expect(workflow["run-name"]).toContain("github.event.pull_request.number"); + expect(workflow["run-name"]).toContain("github.event.pull_request.head.sha"); + expect(workflow["run-name"]).toContain("github.event.pull_request.base.sha"); + expect(workflow["run-name"]).toContain("github.event.changes.base != null"); expect(workflow.on).toEqual({ workflow_run: { workflows: ["CI / Pull Request"], diff --git a/test/pr-workflow-contract.test.ts b/test/pr-workflow-contract.test.ts index 23b0246c90..77d0855470 100644 --- a/test/pr-workflow-contract.test.ts +++ b/test/pr-workflow-contract.test.ts @@ -276,6 +276,7 @@ function codeFilterMatchesChangedPaths(workflow: CiWorkflow, paths: string[]): b describe("pull request and main workflow contracts", () => { const prWorkflow = readYaml(".github/workflows/pr.yaml"); const mainWorkflow = readYaml(".github/workflows/main.yaml"); + const dcoWorkflow = readYaml(".github/workflows/dco-check.yaml"); const installerHashWorkflow = readYaml(".github/workflows/installer-hash-check.yaml"); const installerHashAction = readYaml( ".github/actions/ci-installer-hash-check/action.yaml", @@ -299,6 +300,35 @@ describe("pull request and main workflow contracts", () => { ".github/actions/ci-installer-integration/action.yaml", ), }; + + // source-shape-contract: security -- Base retargets must rerun trusted installer verification without minting skipped required evidence + it("reruns installer hash verification after a pull request base retarget", () => { + expect(installerHashWorkflow.on?.pull_request?.types).toEqual([ + "opened", + "synchronize", + "reopened", + "edited", + ]); + expect(installerHashWorkflow.jobs["check-hash"].if).toBe( + "github.repository == 'NVIDIA/NemoClaw'", + ); + }); + + // source-shape-contract: security -- Dependabot's bounded DCO exemption must report an explicit successful required check + it("records the Dependabot DCO bypass as a successful required job", () => { + const job = dcoWorkflow.jobs["dco-check"]; + const bypass = requiredWorkflowStep(job, "Check Dependabot DCO bypass"); + const declaration = requiredWorkflowStep(job, "Check PR body for Signed-off-by"); + + expect(job.if).toBeUndefined(); + expect(job.steps?.some((step) => step.uses?.startsWith("actions/checkout@"))).toBe(false); + expect(bypass.env?.USERNAME).toBe("${{ github.event.pull_request.user.login }}"); + expect(bypass.run).toContain('"$USERNAME" == "dependabot[bot]"'); + expect(bypass.run).toContain('"$USERNAME" == "app/dependabot"'); + expect(bypass.run).not.toContain(".github/dco-bypass.txt"); + expect(declaration.if).toBe("${{ steps.dco-bypass.outputs.bypass != 'true' }}"); + }); + // source-shape-contract: security -- Installer hashes must be verified by base-trusted or immutable bootstrap code it("runs pull request installer verification from immutable trusted code", () => { const job = installerHashWorkflow.jobs["check-hash"]; @@ -338,6 +368,18 @@ describe("pull request and main workflow contracts", () => { ); expect(installerHashWorkflow.on?.pull_request?.paths).toBeUndefined(); + expect(installerHashWorkflow.on?.pull_request?.types).toEqual([ + "opened", + "synchronize", + "reopened", + "edited", + ]); + expect(installerHashWorkflow["run-name"]).toContain( + "Installer Hash PR #{0} head {1} base {2} gate true", + ); + expect(installerHashWorkflow["run-name"]).toContain("github.event.pull_request.base.sha"); + expect(installerHashWorkflow["run-name"]).not.toContain("github.event.changes.base"); + expect(job.if).toBe("github.repository == 'NVIDIA/NemoClaw'"); expect(installerHashWorkflow.permissions).toEqual({ contents: "read" }); expect(parserRuntimeSetup.uses).toBe(trustedSetupNodeAction); expect(parserRuntimeSetup.with?.["node-version"]).toBe("22.19.0"); diff --git a/test/skills/check-gates-compliance.test.ts b/test/skills/check-gates-compliance.test.ts index d7f20a69c0..5698d76591 100644 --- a/test/skills/check-gates-compliance.test.ts +++ b/test/skills/check-gates-compliance.test.ts @@ -1,228 +1,9 @@ // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. // SPDX-License-Identifier: Apache-2.0 -import { spawnSync } from "node:child_process"; -import fs from "node:fs"; -import os from "node:os"; -import path from "node:path"; - import { describe, expect, it } from "vitest"; -const REQUIRED_CHECK_NAMES = [ - "checks", - "check-hash", - "changes", - "commit-lint", - "dco-check", - "E2E / PR Gate", -] as const; - -interface ComplianceFixture { - body: string; - checkConclusions?: Record; - checkNames?: string[]; - commitOutput?: string; - commitAuthorLogins?: string[]; - contributorCommitPages?: Array< - Array<{ authors: Array<{ login: string }>; authorCount?: number }> - >; - contributorReviewPages?: Array< - Array<{ - author: { login: string }; - state: string; - submittedAt?: string | null; - }> - >; - contributorCommitTotalCount?: number; - contributorReviewTotalCount?: number; - reviews?: Array<{ - author: { login: string }; - state: string; - submittedAt?: string | null; - }>; - prAuthorLogin?: string; - verified: boolean; - reason?: string; -} - -interface ComparatorFixture extends ComplianceFixture { - headRefOid?: string; - state?: string; - mergeable?: string; - mergeStateStatus?: string; - reviewDecision?: string; -} - -function shellSingleQuote(value: string): string { - return `'${value.replaceAll("'", `'"'"'`)}'`; -} - -function runGate(fixture: ComplianceFixture) { - const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "check-gates-compliance-")); - const bin = path.join(tmp, "bin"); - fs.mkdirSync(bin); - const ghPath = path.join(bin, "gh"); - - const pr = { - number: 42, - title: "fix(policy): align maintainer workflow", - url: "https://github.com/NVIDIA/NemoClaw/pull/42", - body: fixture.body, - files: [], - statusCheckRollup: (fixture.checkNames ?? REQUIRED_CHECK_NAMES).map((name) => ({ - __typename: "CheckRun", - name, - status: "COMPLETED", - conclusion: fixture.checkConclusions?.[name] ?? "SUCCESS", - })), - mergeStateStatus: "CLEAN", - headRefOid: "abc123", - author: { login: fixture.prAuthorLogin ?? "contributor" }, - }; - const contributorCommitPages = ( - fixture.contributorCommitPages ?? [ - [ - { - authors: (fixture.commitAuthorLogins ?? ["contributor"]).map((login) => ({ - login, - })), - }, - ], - ] - ).map((page) => - page.map((commit) => ({ - ...commit, - authorCount: commit.authorCount ?? commit.authors.length, - })), - ); - const contributorReviewPages = fixture.contributorReviewPages ?? [ - fixture.reviews ?? [ - { - author: { login: "reviewer" }, - state: "APPROVED", - submittedAt: "2026-01-01T00:00:00Z", - }, - ], - ]; - const contributorCommitOutput = contributorCommitPages - .map((page) => - JSON.stringify({ - nodes: page, - totalCount: fixture.contributorCommitTotalCount ?? contributorCommitPages.flat().length, - }), - ) - .join("\n"); - const contributorReviewOutput = contributorReviewPages - .map((page) => - JSON.stringify({ - nodes: page, - totalCount: fixture.contributorReviewTotalCount ?? contributorReviewPages.flat().length, - }), - ) - .join("\n"); - const commit = { - sha: "abc123", - verified: fixture.verified, - reason: fixture.reason ?? (fixture.verified ? "valid" : "unsigned"), - }; - const commitOutput = fixture.commitOutput ?? JSON.stringify(commit); - - fs.writeFileSync( - ghPath, - `#!/usr/bin/env bash -set -euo pipefail -case "$*" in - "pr view"*) printf '%s' ${shellSingleQuote(JSON.stringify(pr))} ;; - *"ContributorCommits"*) printf '%s' ${shellSingleQuote(contributorCommitOutput)} ;; - *"ContributorReviews"*) printf '%s' ${shellSingleQuote(contributorReviewOutput)} ;; - "api graphql"*) printf '%s' '{"data":{"repository":{"pullRequest":{"reviewThreads":{"nodes":[]}}}}}' ;; - "api repos/NVIDIA/NemoClaw/issues/42/comments"*) printf '%s' '{"id":1,"body":"ordinary comment","user":{"login":"reviewer"},"updated_at":"2026-01-01T00:00:00Z"}' ;; - "api repos/NVIDIA/NemoClaw/pulls/42/commits"*) printf '%s' ${shellSingleQuote(commitOutput)} ;; - *) echo "unexpected gh args: $*" >&2; exit 9 ;; -esac -`, - ); - fs.chmodSync(ghPath, 0o755); - - try { - return spawnSync( - process.execPath, - [ - "--experimental-strip-types", - "--no-warnings", - ".agents/skills/nemoclaw-maintainer-day/scripts/check-gates.ts", - "42", - ], - { - cwd: process.cwd(), - encoding: "utf-8", - env: { ...process.env, PATH: `${bin}${path.delimiter}${process.env.PATH ?? ""}` }, - }, - ); - } finally { - fs.rmSync(tmp, { recursive: true, force: true }); - } -} - -function runComparatorGate(fixture: ComparatorFixture, prNumber = "42") { - const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "collect-gates-compliance-")); - const bin = path.join(tmp, "bin"); - fs.mkdirSync(bin); - const ghPath = path.join(bin, "gh"); - - const pr = { - number: 42, - state: fixture.state ?? "OPEN", - body: fixture.body, - headRefOid: fixture.headRefOid ?? "abc123", - statusCheckRollup: (fixture.checkNames ?? REQUIRED_CHECK_NAMES).map((name) => ({ - name, - status: "COMPLETED", - conclusion: fixture.checkConclusions?.[name] ?? "SUCCESS", - })), - mergeable: fixture.mergeable ?? "MERGEABLE", - mergeStateStatus: fixture.mergeStateStatus ?? "CLEAN", - reviewDecision: fixture.reviewDecision ?? "APPROVED", - }; - const commit = { - sha: "abc123", - verified: fixture.verified, - reason: fixture.reason ?? (fixture.verified ? "valid" : "unsigned"), - }; - const commitOutput = fixture.commitOutput ?? JSON.stringify(commit); - - fs.writeFileSync( - ghPath, - `#!/usr/bin/env bash -set -euo pipefail -case "$1 $2" in - "pr view") printf '%s' ${shellSingleQuote(JSON.stringify(pr))} ;; - "api repos/NVIDIA/NemoClaw/pulls/42/commits") printf '%s' ${shellSingleQuote(commitOutput)} ;; - *) echo "unexpected gh args: $*" >&2; exit 9 ;; -esac -`, - ); - fs.chmodSync(ghPath, 0o755); - - try { - return spawnSync( - "bash", - [ - ".agents/skills/nemoclaw-maintainer-pr-comparator/scripts/collect-gates.sh", - prNumber, - "--repo", - "NVIDIA/NemoClaw", - ], - { - cwd: process.cwd(), - encoding: "utf-8", - env: { ...process.env, PATH: `${bin}${path.delimiter}${process.env.PATH ?? ""}` }, - }, - ); - } finally { - fs.rmSync(tmp, { recursive: true, force: true }); - } -} +import { REQUIRED_CHECK_NAMES, runComparatorGate, runGate } from "./check-gates-test-fixtures.ts"; describe("maintainer merge-gate contributor compliance", () => { it("passes when the PR body has DCO and every commit is GitHub Verified", () => { @@ -231,7 +12,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.gates.contributorCompliance).toMatchObject({ pass: true, @@ -247,9 +27,7 @@ describe("maintainer merge-gate contributor compliance", () => { "not proof of independent approval", ); expect(output.gates).not.toHaveProperty("prAdvisor"); - expect(output.allPass).toBe(true); }); - it("warns without blocking when a contributor also approved (#6222)", () => { const result = runGate({ body: "Signed-off-by: Example User ", @@ -270,7 +48,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", @@ -278,9 +55,7 @@ describe("maintainer merge-gate contributor compliance", () => { uncertainActors: [], }); expect(output.advisories.contributorApprovalOverlap.details).toContain("advisory"); - expect(output.allPass).toBe(true); }); - it("warns when the PR opener approved their own PR (#6222)", () => { const result = runGate({ body: "Signed-off-by: Example User ", @@ -296,16 +71,13 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", actors: ["opener"], uncertainActors: [], }); - expect(output.allPass).toBe(true); }); - it("uses contributors and approvals from every paginated GitHub page (#6222)", () => { const result = runGate({ body: "Signed-off-by: Example User ", @@ -332,16 +104,13 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", actors: ["later-page-contributor"], uncertainActors: [], }); - expect(output.allPass).toBe(true); }); - it("uses a later review page to supersede an earlier approval (#6222)", () => { const result = runGate({ body: "Signed-off-by: Example User ", @@ -365,16 +134,13 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.advisories.contributorApprovalOverlap).toMatchObject({ status: "clear", actors: [], uncertainActors: [], }); - expect(output.allPass).toBe(true); }); - it("warns when a commit author page is incomplete (#6222)", () => { const result = runGate({ body: "Signed-off-by: Example User ", @@ -389,7 +155,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", @@ -399,7 +164,6 @@ describe("maintainer merge-gate contributor compliance", () => { expect(output.advisories.contributorApprovalOverlap.details).toContain( "complete paginated commit and review history", ); - expect(output.allPass).toBe(true); }); it("warns when the paginated review count is incomplete (#6222)", () => { @@ -419,7 +183,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", @@ -429,7 +192,6 @@ describe("maintainer merge-gate contributor compliance", () => { expect(output.advisories.contributorApprovalOverlap.details).toContain( "complete paginated commit and review history", ); - expect(output.allPass).toBe(true); }); it("matches multiple commit authors and co-authors case-insensitively (#6222)", () => { @@ -452,7 +214,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", @@ -486,7 +247,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.advisories.contributorApprovalOverlap).toMatchObject({ status: "clear", @@ -514,7 +274,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); expect(JSON.parse(result.stdout).advisories.contributorApprovalOverlap).toMatchObject({ status: "clear", actors: [], @@ -541,7 +300,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); expect(JSON.parse(result.stdout).advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", actors: ["contributor"], @@ -568,7 +326,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); expect(JSON.parse(result.stdout).advisories.contributorApprovalOverlap).toMatchObject({ status: "clear", actors: [], @@ -590,7 +347,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const advisory = JSON.parse(result.stdout).advisories.contributorApprovalOverlap; expect(advisory).toMatchObject({ status: "warning", @@ -613,7 +369,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const advisory = JSON.parse(result.stdout).advisories.contributorApprovalOverlap; expect(advisory).toMatchObject({ status: "warning", @@ -642,7 +397,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); expect(JSON.parse(result.stdout).advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", actors: [], @@ -669,7 +423,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); expect(JSON.parse(result.stdout).advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", actors: [], @@ -696,7 +449,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); expect(JSON.parse(result.stdout).advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", actors: [], @@ -723,7 +475,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); expect(JSON.parse(result.stdout).advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", actors: [], @@ -755,7 +506,6 @@ describe("maintainer merge-gate contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); expect(JSON.parse(result.stdout).advisories.contributorApprovalOverlap).toMatchObject({ status: "warning", actors: ["fractional", "offset", "whole-second"], @@ -766,11 +516,46 @@ describe("maintainer merge-gate contributor compliance", () => { it("fails closed when the PR body lacks the DCO declaration", () => { const result = runGate({ body: "## Summary\n\nNo declaration.", verified: true }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.gates.contributorCompliance.pass).toBe(false); expect(output.gates.contributorCompliance.details).toContain("lacks a valid Signed-off-by"); - expect(output.allPass).toBe(false); + }); + + it.each([ + "app/dependabot", + "dependabot[bot]", + ])("accepts the explicit PR-body DCO bypass for %s", (prAuthorLogin) => { + const output = JSON.parse( + runGate({ + body: "Automated dependency update.", + prAuthorLogin, + verified: true, + }).stdout, + ); + + expect(output.gates.contributorCompliance).toMatchObject({ + pass: true, + dcoDeclarationPresent: false, + dcoDeclarationBypassed: true, + unverifiedCommits: [], + }); + }); + + it("still rejects an unverified Dependabot commit", () => { + const output = JSON.parse( + runGate({ + body: "Automated dependency update.", + prAuthorLogin: "app/dependabot", + verified: false, + reason: "unsigned", + }).stdout, + ); + + expect(output.gates.contributorCompliance).toMatchObject({ + pass: false, + dcoDeclarationBypassed: true, + unverifiedCommits: [{ sha: "abc123", reason: "unsigned" }], + }); }); it("fails closed when any PR commit is not GitHub Verified", () => { @@ -780,14 +565,12 @@ describe("maintainer merge-gate contributor compliance", () => { reason: "unsigned", }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.gates.contributorCompliance).toMatchObject({ pass: false, dcoDeclarationPresent: true, unverifiedCommits: [{ sha: "abc123", reason: "unsigned" }], }); - expect(output.allPass).toBe(false); }); it("fails closed for type-skewed commit verification data", () => { @@ -801,13 +584,11 @@ describe("maintainer merge-gate contributor compliance", () => { }), }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.gates.contributorCompliance).toMatchObject({ pass: false, unverifiedCommits: [{ sha: "abc123", reason: "malformed_commit_verification_data" }], }); - expect(output.allPass).toBe(false); }); }); @@ -818,7 +599,6 @@ describe("maintainer PR comparator contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.gates.ci_green_latest_sha).toBe(true); expect(output.gates.contributor_compliance).toBe(true); @@ -829,6 +609,41 @@ describe("maintainer PR comparator contributor compliance", () => { }); }); + it.each([ + "app/dependabot", + "dependabot[bot]", + ])("accepts the explicit PR-body DCO bypass for %s", (prAuthorLogin) => { + const result = runComparatorGate({ + body: "Automated dependency update.", + prAuthorLogin, + verified: true, + }); + + const output = JSON.parse(result.stdout); + expect(output.gates.contributor_compliance).toBe(true); + expect(output.details).toMatchObject({ + dco_declaration_present: false, + dco_declaration_bypassed: true, + unverified_commits: [], + }); + }); + + it("still rejects an unverified Dependabot commit", () => { + const result = runComparatorGate({ + body: "Automated dependency update.", + prAuthorLogin: "app/dependabot", + verified: false, + reason: "unsigned", + }); + + const output = JSON.parse(result.stdout); + expect(output.gates.contributor_compliance).toBe(false); + expect(output.details).toMatchObject({ + dco_declaration_bypassed: true, + unverified_commits: [{ sha: "abc123", reason: "unsigned" }], + }); + }); + it("fails when a commit is not verified", () => { const result = runComparatorGate({ body: "Signed-off-by: Example User ", @@ -836,7 +651,6 @@ describe("maintainer PR comparator contributor compliance", () => { reason: "unsigned", }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.gates.contributor_compliance).toBe(false); expect(output.details.unverified_commits).toEqual([{ sha: "abc123", reason: "unsigned" }]); @@ -850,7 +664,6 @@ describe("maintainer PR comparator contributor compliance", () => { commitOutput: "not-json", }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.gates.contributor_compliance).toBe(false); expect(output.details).toMatchObject({ @@ -868,7 +681,6 @@ describe("maintainer PR comparator contributor compliance", () => { verified: true, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.gates.contributor_compliance).toBe(false); expect(output.details.dco_declaration_present).toBe(false); @@ -884,7 +696,6 @@ describe("maintainer PR comparator contributor compliance", () => { '42,"injected":true', ); - expect(result.status).toBe(0); expect(JSON.parse(result.stdout)).toEqual({ pr: '42,"injected":true', error: "invalid_pr_number", @@ -892,6 +703,19 @@ describe("maintainer PR comparator contributor compliance", () => { expect(result.stderr).toBe(""); }); + it("uses the requested PR number in comparator GitHub fixtures", () => { + const result = runComparatorGate( + { + body: "Signed-off-by: Example User ", + verified: true, + }, + "73", + ); + + expect(JSON.parse(result.stdout).pr).toBe(73); + expect(result.stderr).toBe(""); + }); + it("serializes unusual GitHub string values as valid JSON", () => { const result = runComparatorGate({ body: "Signed-off-by: Example User ", @@ -903,7 +727,6 @@ describe("maintainer PR comparator contributor compliance", () => { reviewDecision: 'APPROVED"unexpected', }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.head_sha).toBe('abc"123\\nnext'); expect(output.details).toMatchObject({ @@ -921,7 +744,6 @@ describe("maintainer PR comparator contributor compliance", () => { checkNames: [], }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.gates.ci_green_latest_sha).toBe(false); expect(output.details.ci_missing_required_checks).toEqual(REQUIRED_CHECK_NAMES); @@ -1007,7 +829,6 @@ describe("maintainer PR comparator contributor compliance", () => { checkConclusions: { checks: conclusion }, }); - expect(result.status).toBe(0); const output = JSON.parse(result.stdout); expect(output.gates.ci_green_latest_sha).toBe(false); expect(output.details.ci_failing_checks).toEqual([`checks: ${conclusion}`]); diff --git a/test/skills/check-gates-evidence.test.ts b/test/skills/check-gates-evidence.test.ts new file mode 100644 index 0000000000..433ca7a135 --- /dev/null +++ b/test/skills/check-gates-evidence.test.ts @@ -0,0 +1,1465 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { describe, expect, it } from "vitest"; +import type { ActionJobFixture, ActionRunFixture } from "./check-gates-test-fixtures.ts"; +import { + BASE_SHA, + CUSTOM_RUN_URL, + coordinationCheck, + E2E_COORDINATION_EXTERNAL_ID, + e2eChecks, + e2eGateCheck, + e2eJobs, + e2eRunFixture, + exactDiffGateRun, + HEAD_SHA, + INCOMPLETE_E2E, + installerHashRun, + prWorkflowJobs, + prWorkflowRun, + REQUIRED_CHECK_NAMES, + runGate, + successfulRequiredChecks, + successfulRequiredChecksWithoutE2e, +} from "./check-gates-test-fixtures.ts"; + +describe("maintainer merge-gate contributor compliance", () => { + it("treats a mergeable PR blocked on required review as conflict-free", () => { + const result = runGate({ + body: "Signed-off-by: Example User ", + verified: true, + mergeable: "MERGEABLE", + mergeStateStatus: "BLOCKED", + }); + + const output = JSON.parse(result.stdout); + expect(output.gates.conflicts).toMatchObject({ + pass: true, + mergeable: "MERGEABLE", + mergeStateStatus: "BLOCKED", + }); + expect(output.allPass).toBe(true); + }); + + it("fails closed when BLOCKED masks a stale base revision", () => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + mergeable: "MERGEABLE", + mergeStateStatus: "BLOCKED", + currentBaseSha: "cccccccccccccccccccccccccccccccccccccccc", + }).stdout, + ); + + expect(output.gates.conflicts).toMatchObject({ + pass: false, + mergeable: "MERGEABLE", + mergeStateStatus: "BLOCKED", + baseSha: BASE_SHA, + currentBaseSha: "cccccccccccccccccccccccccccccccccccccccc", + }); + expect(output.allPass).toBe(false); + }); + + it("fails closed when the current base revision cannot be verified", () => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + currentBaseSha: null, + }).stdout, + ); + + expect(output.gates.conflicts).toMatchObject({ + pass: false, + baseSha: BASE_SHA, + }); + expect(output.gates.conflicts.currentBaseSha).toBeUndefined(); + expect(output.allPass).toBe(false); + }); + + it("fails closed while GitHub has not determined mergeability", () => { + const result = runGate({ + body: "Signed-off-by: Example User ", + verified: true, + mergeable: "UNKNOWN", + mergeStateStatus: "UNKNOWN", + }); + + const output = JSON.parse(result.stdout); + expect(output.gates.conflicts).toMatchObject({ + pass: false, + mergeable: "UNKNOWN", + mergeStateStatus: "UNKNOWN", + }); + expect(output.allPass).toBe(false); + }); + + it("fails closed when the PR branch is behind its base branch", () => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + mergeable: "MERGEABLE", + mergeStateStatus: "BEHIND", + }).stdout, + ); + + expect(output.gates.conflicts).toMatchObject({ + pass: false, + mergeable: "MERGEABLE", + mergeStateStatus: "BEHIND", + }); + expect(output.allPass).toBe(false); + }); + + it.each([ + ["title", { title: "fix(policy): changed during review" }], + ["body", { body: "DCO declaration removed during review" }], + ["head", { headRefOid: "c".repeat(40) }], + ["base", { baseRefOid: "c".repeat(40) }], + ["base branch", { baseRefName: "release" }], + ["mergeability", { mergeable: "UNKNOWN" }], + ["merge state", { mergeStateStatus: "UNKNOWN" }], + ])("fails closed when the PR %s changes during gate evaluation", (_name, finalPr) => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + finalPr, + }).stdout, + ); + + expect(output.gates.conflicts).toMatchObject({ + pass: false, + details: "PR revision or merge state changed during gate evaluation; rerun the gate checker", + }); + expect(output.allPass).toBe(false); + }); + + it("makes the PR revision snapshot the final remote read", () => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + finalPrAfterCurrentBase: { headRefOid: "c".repeat(40) }, + }).stdout, + ); + + expect(output.gates.conflicts).toMatchObject({ + pass: false, + details: "PR revision or merge state changed during gate evaluation; rerun the gate checker", + }); + expect(output.allPass).toBe(false); + }); + + it.each([ + ["closes", { state: "CLOSED" }, "PR is no longer open"], + ["becomes a draft", { isDraft: true }, "PR became a draft during gate evaluation"], + ])("fails closed when the PR %s", (_name, finalPr, details) => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + finalPr, + }).stdout, + ); + + expect(output.gates.conflicts).toMatchObject({ pass: false, details }); + expect(output.allPass).toBe(false); + }); + + it("requires checks and changes to come from the same substantive PR CI run", () => { + const statusChecks = successfulRequiredChecks().map((check) => + check.name === "changes" + ? e2eGateCheck([95, 1, "SUCCESS", undefined, undefined, "CI / Pull Request", "changes"]) + : check, + ); + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks, + actionRunAttempts: { + "95": prWorkflowRun("success", [{ id: 1, name: "changes" }], true), + }, + }).stdout, + ); + + expect(output.gates.ci).toMatchObject({ pass: false }); + expect(output.gates.ci.failingChecks).toEqual( + expect.arrayContaining([ + "checks: latest attempt evidence incomplete", + "changes: latest attempt evidence incomplete", + ]), + ); + }); + + it.each([ + [ + "a stale base in the immutable PR CI title", + `CI PR #42 head ${HEAD_SHA} base ${"c".repeat(40)} gate true`, + ], + [ + "a stale head in the immutable PR CI title", + `CI PR #42 head ${"c".repeat(40)} base ${BASE_SHA} gate true`, + ], + [ + "an unsafe PR number in the immutable PR CI title", + `CI PR #9007199254740993 head ${HEAD_SHA} base ${BASE_SHA} gate true`, + ], + [ + "an uppercase SHA in the immutable PR CI title", + `CI PR #42 head ${HEAD_SHA.toUpperCase()} base ${BASE_SHA} gate true`, + ], + ])("rejects %s even when mutable pull_requests claims the current diff", (_name, displayTitle) => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + actionRunAttempts: { + "90": { + ...prWorkflowRun( + "success", + [ + { id: 1, name: "checks" }, + { id: 2, name: "changes" }, + ], + true, + ), + displayTitle, + }, + }, + }).stdout, + ); + + expect(output.gates.ci).toMatchObject({ pass: false }); + expect(output.gates.ci.failingChecks).toEqual( + expect.arrayContaining([ + "checks: latest attempt evidence incomplete", + "changes: latest attempt evidence incomplete", + ]), + ); + }); + + it("fails closed when immutable PR CI identity changes during job collection", () => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + actionRunAttempts: { + "90": { + ...prWorkflowRun( + "success", + [ + { id: 1, name: "checks" }, + { id: 2, name: "changes" }, + ], + true, + ), + nextDisplayTitle: `CI PR #42 head ${HEAD_SHA} base ${"c".repeat(40)} gate true`, + }, + }, + }).stdout, + ); + + expect(output.gates.ci).toMatchObject({ pass: false }); + expect(output.gates.ci.failingChecks).toEqual( + expect.arrayContaining([ + "checks: latest attempt evidence incomplete", + "changes: latest attempt evidence incomplete", + ]), + ); + }); + + it("rejects a required check emitted by the wrong workflow", () => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + actionRunAttempts: { + "91": { + ...exactDiffGateRun("success", [{ id: 1, name: "check-hash" }]), + event: "pull_request", + path: ".github/workflows/unrelated.yaml", + }, + }, + }).stdout, + ); + + expect(output.gates.ci).toMatchObject({ + pass: false, + failingChecks: ["check-hash: latest attempt evidence incomplete"], + }); + }); + + it("accepts immutable installer identity without relying on retarget timestamps", () => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + actionRunAttempts: { + "91": { + ...installerHashRun("success", [{ id: 1, name: "check-hash" }], true), + createdAt: "2026-01-01T00:02:00Z", + updatedAt: "2026-01-01T00:02:00Z", + pullRequests: [], + }, + }, + }).stdout, + ); + + expect(output).toMatchObject({ allPass: true, gates: { ci: { pass: true } } }); + }); + + it.each(["SKIPPED", "NEUTRAL"])("rejects a required Actions run concluded %s", (conclusion) => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + actionRunAttempts: { + "91": installerHashRun(conclusion.toLowerCase(), [{ id: 1, name: "check-hash" }], true), + }, + }).stdout, + ); + + expect(output.gates.ci).toMatchObject({ + pass: false, + failingChecks: ["check-hash: latest attempt evidence incomplete"], + }); + expect(output.allPass).toBe(false); + }); + + it.each(["SKIPPED", "NEUTRAL"])("rejects a required CheckRun concluded %s", (conclusion) => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: successfulRequiredChecks().map((check) => + check.name === "check-hash" ? { ...check, conclusion } : check, + ), + }).stdout, + ); + + expect(output.gates.ci).toMatchObject({ + pass: false, + failingChecks: [`check-hash: ${conclusion}`], + }); + expect(output.allPass).toBe(false); + }); + + it.each([ + ["a stale base", `Installer Hash PR #42 head ${HEAD_SHA} base ${"c".repeat(40)} gate true`], + ["a stale head", `Installer Hash PR #42 head ${"c".repeat(40)} base ${BASE_SHA} gate true`], + [ + "an unsafe PR number", + `Installer Hash PR #9007199254740993 head ${HEAD_SHA} base ${BASE_SHA} gate true`, + ], + [ + "an uppercase SHA", + `Installer Hash PR #42 head ${HEAD_SHA.toUpperCase()} base ${BASE_SHA} gate true`, + ], + ["a malformed title", "Installer Hash current diff"], + ])("rejects check-hash with %s in its immutable title", (_name, displayTitle) => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + actionRunAttempts: { + "91": { + ...installerHashRun("success", [{ id: 1, name: "check-hash" }], true), + displayTitle, + }, + }, + }).stdout, + ); + + expect(output.gates.ci).toMatchObject({ + pass: false, + failingChecks: ["check-hash: latest attempt evidence incomplete"], + }); + }); + + it("does not accept a gate-false installer metadata edit as evidence", () => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + actionRunAttempts: { + "91": installerHashRun("success", [{ id: 1, name: "check-hash" }], false), + }, + }).stdout, + ); + + expect(output.gates.ci).toMatchObject({ + pass: false, + failingChecks: ["check-hash: latest attempt evidence incomplete"], + }); + }); + + it("rejects installer evidence with a contradictory mutable PR association", () => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + actionRunAttempts: { + "91": { + ...installerHashRun("success", [{ id: 1, name: "check-hash" }], true), + pullRequests: [ + { + number: 99, + head: { sha: HEAD_SHA }, + base: { sha: BASE_SHA }, + }, + ], + }, + }, + }).stdout, + ); + + expect(output.gates.ci).toMatchObject({ + pass: false, + failingChecks: ["check-hash: latest attempt evidence incomplete"], + }); + }); + + it("accepts the legacy exact-diff E2E coordination check name", () => { + const legacyCheck = coordinationCheck({ id: 8001, name: "E2E / PR Gate" }); + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [ + ...successfulRequiredChecksWithoutE2e(), + { + __typename: "CheckRun", + name: "E2E / PR Gate", + workflowName: "Automation / Request NVSkills CI", + detailsUrl: "https://github.com/NVIDIA/NemoClaw/runs/8001", + startedAt: "2026-01-01T00:00:00Z", + status: "COMPLETED", + conclusion: "SUCCESS", + }, + ], + coordinationCheckPages: [{ total_count: 0, check_runs: [] }], + legacyCoordinationCheckPages: [{ total_count: 1, check_runs: [legacyCheck] }], + }).stdout, + ); + + expect(output).toMatchObject({ allPass: true, gates: { ci: { pass: true } } }); + }); + + it("finds the exact E2E coordination check on a later page", () => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + coordinationCheckPages: [ + { + total_count: 2, + check_runs: [coordinationCheck({ id: 8001, external_id: "ordinary-uuid" })], + }, + { total_count: 2, check_runs: [coordinationCheck({ id: 8002 })] }, + ], + }).stdout, + ); + + expect(output).toMatchObject({ allPass: true, gates: { ci: { pass: true } } }); + }); + + it.each([ + ["missing", [{ total_count: 0, check_runs: [] }]], + [ + "bound to another diff", + [ + { + total_count: 1, + check_runs: [coordinationCheck({ external_id: `${E2E_COORDINATION_EXTERNAL_ID}-stale` })], + }, + ], + ], + [ + "reported on another head SHA", + [{ total_count: 1, check_runs: [coordinationCheck({ head_sha: "c".repeat(40) })] }], + ], + [ + "claimed by another GitHub App", + [{ total_count: 1, check_runs: [coordinationCheck({ app: { id: 1234 } })] }], + ], + [ + "still running", + [ + { + total_count: 1, + check_runs: [coordinationCheck({ status: "in_progress", conclusion: null })], + }, + ], + ], + [ + "completed with failure", + [ + { + total_count: 1, + check_runs: [coordinationCheck({ status: "completed", conclusion: "failure" })], + }, + ], + ], + [ + "reported with malformed timing", + [{ total_count: 1, check_runs: [coordinationCheck({ started_at: "not-a-time" })] }], + ], + [ + "reported with inverted timing", + [ + { + total_count: 1, + check_runs: [ + coordinationCheck({ + started_at: "2026-01-01T00:02:30Z", + completed_at: "2026-01-01T00:01:30Z", + }), + ], + }, + ], + ], + [ + "duplicated", + [ + { + total_count: 2, + check_runs: [coordinationCheck(), coordinationCheck({ id: 8001 })], + }, + ], + ], + ["from an incomplete page set", [{ total_count: 2, check_runs: [coordinationCheck()] }]], + ])("fails closed when exact-diff E2E coordination evidence is %s", (_name, pages) => { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + coordinationCheckPages: pages, + }).stdout, + ); + + expect(output.gates.ci).toMatchObject({ + pass: false, + failingChecks: ["E2E / PR Gate: latest attempt evidence incomplete"], + }); + }); + + it("requires exact-diff evidence for optional Actions checks", () => { + const result = runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [ + ...successfulRequiredChecks(), + { + __typename: "CheckRun", + name: "optional-check", + workflowName: "CI / Optional", + detailsUrl: "https://github.com/NVIDIA/NemoClaw/actions/runs/443/job/41", + startedAt: "2026-01-01T00:00:00Z", + status: "COMPLETED", + conclusion: "SUCCESS", + }, + ], + actionRunAttempts: { + "443": { + ...exactDiffGateRun("success", [{ id: 41, name: "optional-check" }]), + headSha: "stale", + pullRequestHeadSha: HEAD_SHA, + }, + }, + }); + + const output = JSON.parse(result.stdout); + expect(output.gates.ci).toMatchObject({ + pass: false, + failingChecks: ["optional-check: latest attempt evidence incomplete"], + }); + expect(output.allPass).toBe(false); + }); + + it.each([ + "push", + "dynamic", + ])("accepts an optional %s check tied to the current head SHA", (event) => { + const result = runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [ + ...successfulRequiredChecks(), + { + __typename: "CheckRun", + name: "optional-check", + workflowName: "CI / Optional", + detailsUrl: "https://github.com/NVIDIA/NemoClaw/actions/runs/446/job/41", + startedAt: "2026-01-01T00:00:00Z", + status: "COMPLETED", + conclusion: "SUCCESS", + }, + ], + actionRunAttempts: { + "446": { + attempt: 1, + headSha: HEAD_SHA, + event, + path: ".github/workflows/optional.yaml", + status: "completed", + conclusion: "success", + jobs: [{ id: 41, name: "optional-check" }], + }, + }, + }); + + expect(JSON.parse(result.stdout)).toMatchObject({ + allPass: true, + gates: { ci: { pass: true } }, + }); + }); + + it("accepts duplicate optional runs with exact-PR and current-head identities", () => { + const optionalCheck = (runId: number, jobId: number, startedAt: string) => ({ + __typename: "CheckRun", + name: "request", + workflowName: "Automation / Request NVSkills CI", + detailsUrl: `https://github.com/NVIDIA/NemoClaw/actions/runs/${runId}/job/${jobId}`, + startedAt, + status: "COMPLETED", + conclusion: "SKIPPED", + }); + const skippedJob = (id: number): ActionJobFixture => ({ + id, + name: "request", + conclusion: "skipped", + }); + const result = runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [ + ...successfulRequiredChecks(), + optionalCheck(447, 41, "2026-01-01T00:00:00Z"), + optionalCheck(448, 42, "2026-01-01T00:02:00Z"), + ], + actionRunAttempts: { + "447": { + ...exactDiffGateRun("skipped", [skippedJob(41)]), + event: "push", + path: ".github/workflows/request-nvskills-ci.yml", + }, + "448": { + attempt: 1, + headSha: HEAD_SHA, + event: "push", + path: ".github/workflows/request-nvskills-ci.yml", + status: "completed", + conclusion: "skipped", + jobs: [skippedJob(42)], + }, + }, + }); + + expect(JSON.parse(result.stdout)).toMatchObject({ + allPass: true, + gates: { ci: { pass: true } }, + }); + }); + + it.each([ + { + name: "workflow-run status", + run: { status: "future_status" }, + }, + { + name: "workflow-run conclusion", + run: { conclusion: "future_conclusion" }, + }, + { + name: "job status", + job: { status: "future_status" }, + }, + { + name: "job conclusion", + job: { conclusion: "future_conclusion" }, + }, + ])("fails closed on an unknown $name", ({ run, job }) => { + const result = runGate( + e2eRunFixture(e2eChecks([444, 41, "SUCCESS"]), { + "444": { + ...exactDiffGateRun("success", [{ id: 41, name: "E2E / PR Gate", ...job }]), + ...run, + }, + }), + ); + + const output = JSON.parse(result.stdout); + expect(output.gates.ci).toMatchObject({ + pass: false, + failingChecks: ["E2E / PR Gate: latest attempt evidence incomplete"], + }); + expect(output.allPass).toBe(false); + }); + + it.each([ + { + name: "still running", + run: { status: "in_progress", conclusion: null }, + jobs: [{ id: 41, name: "E2E / PR Gate" }], + }, + { + name: "failed with a differently named failed job", + run: { status: "completed", conclusion: "failure" }, + jobs: [ + { id: 41, name: "E2E / PR Gate" }, + { id: 42, name: "hidden-failure", conclusion: "failure" }, + ], + }, + ])("fails closed when an Actions run is $name", ({ run, jobs }) => { + const result = runGate( + e2eRunFixture(e2eChecks([445, 41, "SUCCESS"]), { + "445": { + ...exactDiffGateRun("success", jobs), + ...run, + }, + }), + ); + + const output = JSON.parse(result.stdout); + expect(output.gates.ci).toMatchObject({ + pass: false, + failingChecks: ["E2E / PR Gate: latest attempt evidence incomplete"], + }); + expect(output.allPass).toBe(false); + }); + + it("uses the latest attempt for duplicate check-run contexts", () => { + const result = runGate( + e2eRunFixture( + [ + [100, 1, "CANCELLED"], + [101, 2, "SUCCESS"], + ], + { + "100": { + ...exactDiffGateRun("cancelled", [{ id: 1, name: "E2E / PR Gate" }]), + createdAt: "2026-01-01T00:00:00Z", + }, + "101": { + ...exactDiffGateRun("success", [{ id: 2, name: "E2E / PR Gate" }]), + createdAt: "2026-01-01T00:01:00Z", + }, + }, + ), + ); + + const output = JSON.parse(result.stdout); + expect(output.gates.ci).toMatchObject({ pass: true }); + }); + it("orders overlapping workflow runs by run creation time", () => { + const result = runGate( + e2eRunFixture( + [ + [102, 3, "SUCCESS", "2026-01-01T00:03:00Z"], + [103, 4, "FAILURE", "2026-01-01T00:02:00Z"], + ], + { + "102": { + ...exactDiffGateRun("success", [{ id: 3, name: "E2E / PR Gate" }]), + createdAt: "2026-01-01T00:00:00Z", + }, + "103": { + ...exactDiffGateRun("failure", [{ id: 4, name: "E2E / PR Gate" }]), + createdAt: "2026-01-01T00:01:00Z", + }, + }, + ), + ); + + expect(JSON.parse(result.stdout).gates.ci).toMatchObject({ + pass: false, + failingChecks: ["E2E / PR Gate: FAILURE"], + }); + }); + it("keeps every duplicate job from the latest workflow run", () => { + const result = runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [ + ...REQUIRED_CHECK_NAMES.map((name) => ({ + __typename: "CheckRun", + name, + workflowName: `CI / ${name}`, + detailsUrl: `https://github.com/NVIDIA/NemoClaw/actions/runs/200/job/${name}`, + startedAt: "2026-01-01T00:02:00Z", + status: "COMPLETED", + conclusion: "SUCCESS", + })), + { + __typename: "CheckRun", + name: "matrix-check", + workflowName: "CI / Matrix", + detailsUrl: "https://github.com/NVIDIA/NemoClaw/actions/runs/199/job/1", + startedAt: "2026-01-01T00:00:00Z", + status: "COMPLETED", + conclusion: "SUCCESS", + }, + { + __typename: "CheckRun", + name: "matrix-check", + workflowName: "CI / Matrix", + detailsUrl: "https://github.com/NVIDIA/NemoClaw/actions/runs/200/job/2", + startedAt: "2026-01-01T00:02:00Z", + status: "COMPLETED", + conclusion: "SUCCESS", + }, + { + __typename: "CheckRun", + name: "matrix-check", + workflowName: "CI / Matrix", + detailsUrl: "https://github.com/NVIDIA/NemoClaw/actions/runs/200/job/3", + startedAt: "2026-01-01T00:03:00Z", + status: "COMPLETED", + conclusion: "FAILURE", + }, + ], + }); + + const output = JSON.parse(result.stdout); + expect(output.gates.ci).toMatchObject({ + pass: false, + failingChecks: ["matrix-check: FAILURE"], + }); + }); + it("accepts exact-head evidence from a non-PR Actions event", () => { + const fixture = e2eRunFixture(e2eChecks([874, 2, "SUCCESS"]), { + "874": exactDiffGateRun("success", e2eJobs(2)), + "875": { + attempt: 1, + headSha: HEAD_SHA, + event: "dynamic", + path: "dynamic/github-code-scanning/codeql", + status: "completed", + conclusion: "success", + jobs: [{ id: 1, name: "optional-check" }], + }, + }); + fixture.statusChecks?.push( + e2eGateCheck([875, 1, "SUCCESS", undefined, undefined, "CodeQL", "optional-check"]), + ); + expect(JSON.parse(runGate(fixture).stdout).gates.ci).toMatchObject({ pass: true }); + }); + it("rejects required checks represented only by a status context", () => { + const fixture = e2eRunFixture([], {}); + fixture.statusChecks?.push({ + __typename: "StatusContext", + context: "E2E / PR Gate", + state: "SUCCESS", + }); + expect(JSON.parse(runGate(fixture).stdout).gates.ci).toMatchObject({ + pass: false, + failingChecks: INCOMPLETE_E2E, + }); + }); + it("uses the latest attempt for custom check-run details URLs", () => { + const fixture = e2eRunFixture( + [ + [874, 2, "SUCCESS"], + [0, 0, "FAILURE", "2026-01-01T00:00:00Z", `${CUSTOM_RUN_URL}1`, "CodeQL", "custom-check"], + [0, 0, "SUCCESS", "2026-01-01T00:02:00Z", `${CUSTOM_RUN_URL}2`, "CodeQL", "custom-check"], + ], + { "874": exactDiffGateRun("success", e2eJobs(2)) }, + ); + expect(JSON.parse(runGate(fixture).stdout).gates.ci).toMatchObject({ pass: true }); + }); + it("uses the latest attempt when GitHub reuses an Actions run ID", () => { + const fixture = { + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [ + ...successfulRequiredChecksWithoutE2e(), + e2eGateCheck([300, 10, "FAILURE", "2026-01-01T00:00:00Z"]), + e2eGateCheck([300, 20, "SUCCESS", "2026-01-01T00:02:00Z"]), + ], + }; + const result = runGate({ + ...fixture, + actionRunAttempts: { + "300": exactDiffGateRun("success", [{ id: 20, name: "E2E / PR Gate" }], 2), + }, + }); + + const output = JSON.parse(result.stdout); + expect(output.gates.ci).toMatchObject({ pass: true }); + + const unavailable = runGate(fixture); + expect(JSON.parse(unavailable.stdout).gates.ci).toMatchObject({ + pass: false, + failingChecks: ["E2E / PR Gate: FAILURE"], + }); + }); + it("uses an envelope-bound E2E run when a later association-less label run is skipped", () => { + const result = runGate( + e2eRunFixture( + [ + [400, 40, "SUCCESS"], + [401, 41, "SKIPPED"], + ], + { + "400": { + ...exactDiffGateRun("success", [{ id: 40, name: "E2E / PR Gate" }]), + pullRequests: [], + createdAt: "2026-01-01T00:01:00Z", + updatedAt: "2026-01-01T00:03:00Z", + }, + "401": { + ...exactDiffGateRun("skipped", [ + { id: 41, name: "E2E / PR Gate", conclusion: "skipped" }, + ]), + pullRequests: [], + createdAt: "2026-01-01T00:04:00Z", + updatedAt: "2026-01-01T00:05:00Z", + displayTitle: `E2E Gate PR #42 head ${HEAD_SHA} base ${BASE_SHA} gate false`, + }, + }, + ), + ); + + expect(JSON.parse(result.stdout)).toMatchObject({ + allPass: true, + gates: { ci: { pass: true } }, + }); + }); + + it("does not discard a skipped E2E run with malformed immutable identity", () => { + const result = runGate( + e2eRunFixture( + [ + [405, 45, "SUCCESS"], + [406, 46, "SKIPPED"], + ], + { + "405": exactDiffGateRun("success", [{ id: 45, name: "E2E / PR Gate" }]), + "406": { + ...exactDiffGateRun("skipped", [ + { id: 46, name: "E2E / PR Gate", conclusion: "skipped" }, + ]), + displayTitle: "E2E Gate stale metadata", + }, + }, + ), + ); + + expect(JSON.parse(result.stdout).gates.ci).toMatchObject({ + pass: false, + failingChecks: ["E2E / PR Gate: SKIPPED"], + }); + }); + + it.each([ + [ + "does not enclose trusted coordination", + { + createdAt: "2026-01-01T00:02:00Z", + updatedAt: "2026-01-01T00:03:00Z", + }, + ], + [ + "names another PR base", + { + displayTitle: `E2E Gate PR #42 head ${HEAD_SHA} base ${"c".repeat(40)} gate true`, + }, + ], + [ + "names another PR number", + { + displayTitle: `E2E Gate PR #43 head ${HEAD_SHA} base ${BASE_SHA} gate true`, + }, + ], + ["has another head branch", { headBranch: "other-branch" }], + ["has another head repository", { headRepository: "example/fork" }], + ])("rejects an association-less E2E run that %s", (_name, overrides) => { + const result = runGate( + e2eRunFixture(e2eChecks([402, 42, "SUCCESS"]), { + "402": { + ...exactDiffGateRun("success", [{ id: 42, name: "E2E / PR Gate" }]), + pullRequests: [], + ...overrides, + }, + }), + ); + + expect(JSON.parse(result.stdout).gates.ci).toMatchObject({ + pass: false, + failingChecks: INCOMPLETE_E2E, + }); + }); + + it("fails closed when an Actions run timing changes during job collection", () => { + const result = runGate( + e2eRunFixture(e2eChecks([403, 43, "SUCCESS"]), { + "403": { + ...exactDiffGateRun("success", [{ id: 43, name: "E2E / PR Gate" }]), + nextUpdatedAt: "2026-01-01T00:04:00Z", + }, + }), + ); + + expect(JSON.parse(result.stdout).gates.ci).toMatchObject({ + pass: false, + failingChecks: INCOMPLETE_E2E, + }); + }); + + it("fails closed when E2E controller identity changes during job collection", () => { + const result = runGate( + e2eRunFixture(e2eChecks([404, 44, "SUCCESS"]), { + "404": { + ...exactDiffGateRun("success", [{ id: 44, name: "E2E / PR Gate" }]), + nextDisplayTitle: `E2E Gate PR #42 head ${HEAD_SHA} base ${"c".repeat(40)} gate true`, + }, + }), + ); + + expect(JSON.parse(result.stdout).gates.ci).toMatchObject({ + pass: false, + failingChecks: INCOMPLETE_E2E, + }); + }); + it("keeps substantive PR CI ahead of a later metadata-only edit run", () => { + const checkRun = ( + name: string, + runId: number, + jobId: number, + conclusion: string, + startedAt: string, + ) => ({ + __typename: "CheckRun", + name, + workflowName: "CI / Pull Request", + detailsUrl: `https://github.com/NVIDIA/NemoClaw/actions/runs/${runId}/job/${jobId}`, + startedAt, + status: "COMPLETED", + conclusion, + }); + const result = runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [ + ...successfulRequiredChecks().filter((check) => check.name !== "checks"), + checkRun("static-checks", 800, 3, "FAILURE", "2026-01-01T00:00:00Z"), + checkRun("checks", 800, 11, "FAILURE", "2026-01-01T00:00:00Z"), + checkRun("static-checks", 801, 3, "SKIPPED", "2026-01-01T00:02:00Z"), + checkRun("checks", 801, 11, "SUCCESS", "2026-01-01T00:02:00Z"), + ], + actionRunAttempts: { + "800": prWorkflowRun( + "failure", + prWorkflowJobs("success", { + changes: { conclusion: "success" }, + "static-checks": { conclusion: "failure" }, + checks: { conclusion: "failure" }, + }), + true, + ), + "801": prWorkflowRun( + "success", + prWorkflowJobs("skipped", { + checks: { conclusion: "success" }, + changes: { conclusion: "skipped" }, + }), + false, + ), + }, + }); + + const output = JSON.parse(result.stdout); + expect(output.gates.ci.pass).toBe(false); + expect(output.gates.ci.failingChecks).toEqual( + expect.arrayContaining(["static-checks: FAILURE", "checks: FAILURE"]), + ); + expect(output.allPass).toBe(false); + + const invalidShapeResult = runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [ + ...successfulRequiredChecks().filter((check) => check.name !== "checks"), + checkRun("static-checks", 810, 3, "SUCCESS", "2026-01-01T00:00:00Z"), + checkRun("checks", 810, 11, "SUCCESS", "2026-01-01T00:00:00Z"), + checkRun("static-checks", 811, 3, "SKIPPED", "2026-01-01T00:02:00Z"), + checkRun("checks", 811, 11, "SUCCESS", "2026-01-01T00:02:00Z"), + ], + actionRunAttempts: { + "810": prWorkflowRun("success", prWorkflowJobs("success", {}), true), + "811": prWorkflowRun( + "success", + prWorkflowJobs("skipped", { checks: { conclusion: "success" } }).filter( + (job) => job.name !== "plugin-tests", + ), + false, + ), + }, + }); + + const invalidShapeOutput = JSON.parse(invalidShapeResult.stdout); + expect(invalidShapeOutput.gates.ci).toMatchObject({ pass: false }); + expect(invalidShapeOutput.gates.ci.failingChecks).toEqual( + expect.arrayContaining([ + "static-checks: latest attempt evidence incomplete", + "checks: latest attempt evidence incomplete", + ]), + ); + + const unexpectedJobResult = runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [ + ...successfulRequiredChecks().filter((check) => check.name !== "checks"), + checkRun("static-checks", 812, 3, "SUCCESS", "2026-01-01T00:00:00Z"), + checkRun("checks", 812, 11, "SUCCESS", "2026-01-01T00:00:00Z"), + checkRun("static-checks", 813, 3, "SKIPPED", "2026-01-01T00:02:00Z"), + checkRun("checks", 813, 11, "SUCCESS", "2026-01-01T00:02:00Z"), + ], + actionRunAttempts: { + "812": prWorkflowRun("success", prWorkflowJobs("success", {}), true), + "813": prWorkflowRun( + "success", + [ + ...prWorkflowJobs("skipped", { checks: { conclusion: "success" } }), + { id: 12, name: "unexpected-job", conclusion: "success" }, + ], + false, + ), + }, + }); + + expect(JSON.parse(unexpectedJobResult.stdout).gates.ci.failingChecks).toEqual( + expect.arrayContaining([ + "static-checks: latest attempt evidence incomplete", + "checks: latest attempt evidence incomplete", + ]), + ); + }); + + it("fails closed when a gate-false metadata run omits or duplicates a sentinel job", () => { + const metadataJobs = prWorkflowJobs("skipped", { + checks: { conclusion: "success" }, + }); + const malformedRuns = [ + metadataJobs.filter((job) => job.name !== "changes"), + [...metadataJobs, { id: 12, name: "checks", conclusion: "success" }], + ]; + + for (const jobs of malformedRuns) { + const output = JSON.parse( + runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [ + ...successfulRequiredChecks().filter( + (check) => check.name !== "checks" && check.name !== "changes", + ), + e2eGateCheck([90, 11, "SUCCESS", undefined, undefined, "CI / Pull Request", "checks"]), + e2eGateCheck([90, 1, "SUCCESS", undefined, undefined, "CI / Pull Request", "changes"]), + ], + actionRunAttempts: { "90": prWorkflowRun("success", jobs, false) }, + }).stdout, + ); + + expect(output.gates.ci).toMatchObject({ pass: false }); + expect(output.gates.ci.failingChecks).toEqual( + expect.arrayContaining([ + "checks: latest attempt evidence incomplete", + "changes: latest attempt evidence incomplete", + ]), + ); + } + }); + + it("drops an unexpanded metadata-only job when an expanded substantive run exists", () => { + const prWorkflowJobs = (expandedMatrixName = false): ActionJobFixture[] => + [ + "changes", + "docs-only-checks", + "static-checks", + "build-typecheck", + "installer-integration", + "wechat-runtime-audit", + "reviewed-npm-audit", + expandedMatrixName ? "cli-test-shards (1)" : "cli-test-shards", + "cli-tests", + "plugin-tests", + "checks", + ].map((name, index) => ({ id: index + 1, name })); + const checkRun = (name: string, runId: number, jobId: number, conclusion: string) => ({ + __typename: "CheckRun", + name, + workflowName: "CI / Pull Request", + detailsUrl: `https://github.com/NVIDIA/NemoClaw/actions/runs/${runId}/job/${jobId}`, + startedAt: runId === 820 ? "2026-01-01T00:00:00Z" : "2026-01-01T00:02:00Z", + status: "COMPLETED", + conclusion, + }); + const metadataJobs = prWorkflowJobs().map((job) => ({ + ...job, + conclusion: job.name === "checks" ? "success" : "skipped", + })); + const result = runGate({ + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [ + ...successfulRequiredChecks(), + checkRun("cli-test-shards (1)", 820, 8, "SUCCESS"), + checkRun("cli-test-shards", 821, 8, "SKIPPED"), + ], + actionRunAttempts: { + "820": prWorkflowRun("success", prWorkflowJobs(true), true), + "821": prWorkflowRun("success", metadataJobs, false), + }, + }); + + expect(JSON.parse(result.stdout)).toMatchObject({ + allPass: true, + gates: { ci: { pass: true } }, + }); + }); + + it("keeps a later run when only the grouped job was skipped", () => { + const result = runGate( + e2eRunFixture( + [ + [410, 40, "SUCCESS"], + [411, 41, "SKIPPED"], + ], + { + "410": exactDiffGateRun("success", [{ id: 40, name: "E2E / PR Gate" }]), + "411": exactDiffGateRun("success", [ + { id: 41, name: "E2E / PR Gate", conclusion: "skipped" }, + { id: 42, name: "initialize" }, + ]), + }, + ), + ); + + expect(JSON.parse(result.stdout)).toMatchObject({ + allPass: false, + gates: { ci: { pass: false, failingChecks: ["E2E / PR Gate: SKIPPED"] } }, + }); + }); + + it.each([ + { + name: "keeps current-diff evidence ahead of a later nonmatching run", + checks: e2eChecks([420, 40, "FAILURE"], [421, 41, "SUCCESS"]), + runs: { + "420": exactDiffGateRun("failure", [{ id: 40, name: "E2E / PR Gate" }]), + "421": { + ...exactDiffGateRun("success", [{ id: 41, name: "E2E / PR Gate" }]), + headSha: "stale", + }, + } as Record, + failingChecks: ["E2E / PR Gate: FAILURE"], + }, + { + name: "fails closed on a later run with unknown diff identity", + checks: e2eChecks([430, 40, "SUCCESS"], [431, 41, "SUCCESS"]), + runs: { + "430": exactDiffGateRun("success", [{ id: 40, name: "E2E / PR Gate" }]), + "431": { attempt: 1, jobs: [{ id: 41, name: "E2E / PR Gate" }] }, + } as Record, + }, + { + name: "rejects a singleton check from an older run attempt", + checks: e2eChecks([440, 41, "SUCCESS"]), + runs: { + "440": exactDiffGateRun("success", [{ id: 42, name: "E2E / PR Gate" }]), + } as Record, + }, + { + name: "rejects a singleton check from a stale PR diff", + checks: e2eChecks([442, 41, "SUCCESS"]), + runs: { + "442": { + ...exactDiffGateRun("success", e2eJobs(41)), + headSha: "stale", + pullRequestHeadSha: HEAD_SHA, + }, + } as Record, + }, + { + name: "rejects an optional Actions check from a stale PR diff", + checks: e2eChecks( + [442, 41, "SUCCESS"], + [443, 43, "SUCCESS", undefined, undefined, undefined, "optional-check"], + ), + runs: { + "442": exactDiffGateRun("success", e2eJobs(41)), + "443": { + ...exactDiffGateRun("success", [{ id: 43, name: "optional-check" }]), + headSha: "stale", + pullRequestHeadSha: HEAD_SHA, + }, + } as Record, + failingChecks: ["optional-check: latest attempt evidence incomplete"], + }, + { + name: "rejects a singleton Actions check with a malformed URL", + checks: e2eChecks([470, 41, "SUCCESS", undefined, "malformed"]), + runs: {} as Record, + }, + { + name: "rejects a required native check with no workflow or URL identity", + checks: e2eChecks([474, 41, "SUCCESS", undefined, "", ""]), + runs: {} as Record, + }, + { + name: "rejects a required native check with a custom check-run URL", + checks: e2eChecks([475, 41, "SUCCESS", undefined, CUSTOM_RUN_URL, "CodeQL"]), + runs: {} as Record, + }, + { + name: "rejects duplicate Actions checks when one URL is malformed", + checks: e2eChecks([472, 40, "SUCCESS"], [473, 41, "SUCCESS", undefined, "malformed"]), + runs: {} as Record, + }, + { + name: "rejects exact-diff runs with different workflow identities", + checks: e2eChecks([480, 40, "FAILURE"], [481, 41, "SUCCESS"]), + runs: { + "480": { + ...exactDiffGateRun("failure", e2eJobs(40)), + createdAt: "2026-01-01T00:00:00Z", + }, + "481": { + ...exactDiffGateRun("success", e2eJobs(41)), + createdAt: "2026-01-01T00:01:00Z", + path: ".github/workflows/unrelated.yaml", + }, + } as Record, + }, + { + name: "rejects an exact-diff run with a null workflow path", + checks: e2eChecks([482, 41, "SUCCESS"]), + runs: { + "482": { ...exactDiffGateRun("success", e2eJobs(41)), path: undefined }, + } as Record, + }, + { + name: "rejects jobs when a newer run attempt starts during collection", + checks: e2eChecks([490, 41, "SUCCESS"]), + runs: { + "490": { ...exactDiffGateRun("success", e2eJobs(41)), nextAttempt: 2 }, + } as Record, + }, + { + name: "validates latest-attempt jobs for tied workflow runs", + checks: e2eChecks( + [445, 40, "SUCCESS", "2026-01-01T00:00:00Z"], + [446, 41, "SUCCESS", "2026-01-01T00:00:00Z"], + ), + runs: { + "445": exactDiffGateRun("success", [{ id: 42, name: "E2E / PR Gate" }]), + "446": exactDiffGateRun("success", [{ id: 43, name: "E2E / PR Gate" }]), + } as Record, + }, + { + name: "fails closed when prior conclusions are invalid or incomplete", + checks: e2eChecks( + [450, 40, "SUCCESS"], + [452, 42, "SUCCESS"], + [453, 43, "SUCCESS"], + [451, 41, "SKIPPED"], + ), + runs: { + "450": exactDiffGateRun("mystery", e2eJobs(40)), + "452": exactDiffGateRun("success", [ + { id: 42, name: "E2E / PR Gate", conclusion: "mystery" }, + ]), + "453": exactDiffGateRun("success", [{ id: 43, name: "E2E / PR Gate", conclusion: null }]), + "451": exactDiffGateRun("skipped", [ + { id: 41, name: "E2E / PR Gate", conclusion: "skipped" }, + ]), + } as Record, + failingChecks: ["E2E / PR Gate: SKIPPED"], + }, + ])("$name", ({ checks, runs, failingChecks = INCOMPLETE_E2E }) => { + const result = runGate(e2eRunFixture(checks, runs)); + expect(JSON.parse(result.stdout)).toMatchObject({ + allPass: false, + gates: { ci: { pass: false, failingChecks } }, + }); + }); + it("paginates every job before selecting the latest run attempt", () => { + const firstPage = Array.from({ length: 100 }, (_value, index) => ({ + id: index + 20, + name: `unrelated-job-${index}`, + })); + const result = runGate( + e2eRunFixture( + [ + [500, 10, "FAILURE"], + [500, 120, "SUCCESS"], + ], + { + "500": { + ...exactDiffGateRun("success", [], 2), + jobPages: [firstPage, [{ id: 120, name: "E2E / PR Gate" }]], + }, + }, + ), + ); + + expect(JSON.parse(result.stdout)).toMatchObject({ + allPass: true, + gates: { ci: { pass: true } }, + }); + }); + it("fails closed when a latest-attempt job is absent from the PR rollup", () => { + const result = runGate( + e2eRunFixture( + [ + [600, 10, "SUCCESS"], + [600, 20, "SUCCESS"], + ], + { + "600": exactDiffGateRun("success", e2eJobs(20, 21), 2), + }, + ), + ); + + expect(JSON.parse(result.stdout)).toMatchObject({ + allPass: false, + gates: { + ci: { + pass: false, + failingChecks: ["E2E / PR Gate: latest attempt evidence incomplete"], + }, + }, + }); + }); + it.each([ + "", + "1", + "2026-02-30T00:00:00Z", + ])("fails closed on invalid check-run ordering timestamp '%s'", (timestamp) => { + const result = runGate( + e2eRunFixture( + [ + [700, 1, "SUCCESS", timestamp], + [701, 2, "SUCCESS", timestamp], + ], + { + "700": exactDiffGateRun("success", [{ id: 1, name: "E2E / PR Gate" }]), + "701": exactDiffGateRun("success", [{ id: 2, name: "E2E / PR Gate" }]), + }, + ), + ); + + expect(JSON.parse(result.stdout).gates.ci).toMatchObject({ + pass: false, + failingChecks: ["E2E / PR Gate: latest attempt evidence incomplete"], + }); + }); +}); diff --git a/test/skills/check-gates-test-fixtures.ts b/test/skills/check-gates-test-fixtures.ts new file mode 100644 index 0000000000..c1088dc1de --- /dev/null +++ b/test/skills/check-gates-test-fixtures.ts @@ -0,0 +1,568 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; + +const REQUIRED_CHECK_NAMES = [ + "checks", + "check-hash", + "changes", + "commit-lint", + "dco-check", + "E2E / PR Gate", +] as const; + +type E2eCheckFixture = [number, number, string, string?, string?, string?, string?]; +const CUSTOM_RUN_URL = "https://github.com/NVIDIA/NemoClaw/runs/123"; +const INCOMPLETE_E2E = ["E2E / PR Gate: latest attempt evidence incomplete"]; +const HEAD_SHA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"; +const BASE_SHA = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"; +const E2E_COORDINATION_NAME = "E2E / PR Gate Coordination"; +const E2E_COORDINATION_EXTERNAL_ID = `nemoclaw-pr-e2e:v2:42:${HEAD_SHA}:${BASE_SHA}`; +const PR_WORKFLOW_JOB_NAMES = [ + "changes", + "docs-only-checks", + "static-checks", + "build-typecheck", + "installer-integration", + "wechat-runtime-audit", + "reviewed-npm-audit", + "cli-test-shards", + "cli-tests", + "plugin-tests", + "checks", +]; +const REQUIRED_CHECK_RUNS: Record = + { + checks: { runId: 90, jobId: 1, workflowName: "CI / Pull Request" }, + changes: { runId: 90, jobId: 2, workflowName: "CI / Pull Request" }, + "check-hash": { runId: 91, jobId: 1, workflowName: "Security / Installer Hash Check" }, + "commit-lint": { runId: 92, jobId: 1, workflowName: "CI / Commit Lint" }, + "dco-check": { runId: 93, jobId: 1, workflowName: "CI / DCO Check" }, + "E2E / PR Gate": { runId: 94, jobId: 1, workflowName: "E2E / PR Gate Controller" }, + }; + +interface ActionJobFixture { + id: number; + name: string; + status?: string; + conclusion?: string | null; +} + +interface ActionRunFixture { + attempt: number; + nextAttempt?: number; + nextCreatedAt?: string; + nextUpdatedAt?: string; + nextDisplayTitle?: string; + nextStatus?: string; + nextConclusion?: string | null; + jobs?: ActionJobFixture[]; + jobPages?: ActionJobFixture[][]; + createdAt?: string; + updatedAt?: string; + headSha?: string; + headBranch?: string; + headRepository?: string; + pullRequestHeadSha?: string; + pullRequests?: unknown[]; + baseSha?: string; + displayTitle?: string; + event?: string; + path?: string; + status?: string; + conclusion?: string | null; +} + +interface ComplianceFixture { + body: string; + checkConclusions?: Record; + checkNames?: string[]; + statusChecks?: Array<{ + __typename?: string; + name?: string; + context?: string; + workflowName?: string; + startedAt?: string; + completedAt?: string; + detailsUrl?: string; + status?: string; + conclusion?: string; + state?: string; + }>; + commitOutput?: string; + commitAuthorLogins?: string[]; + contributorCommitPages?: Array< + Array<{ authors: Array<{ login: string }>; authorCount?: number }> + >; + contributorReviewPages?: Array< + Array<{ + author: { login: string }; + state: string; + submittedAt?: string | null; + }> + >; + contributorCommitTotalCount?: number; + contributorReviewTotalCount?: number; + reviews?: Array<{ + author: { login: string }; + state: string; + submittedAt?: string | null; + }>; + prAuthorLogin?: string; + mergeable?: string; + mergeStateStatus?: string; + currentBaseSha?: string | null; + verified: boolean; + reason?: string; + actionRunAttempts?: Record; + issueEventPages?: unknown[]; + coordinationCheckPages?: unknown[]; + legacyCoordinationCheckPages?: unknown[]; + finalPr?: Record; + finalPrAfterCurrentBase?: Record; +} + +interface ComparatorFixture extends ComplianceFixture { + headRefOid?: string; + state?: string; + mergeable?: string; + mergeStateStatus?: string; + reviewDecision?: string; +} + +function shellSingleQuote(value: string): string { + return `'${value.replaceAll("'", `'"'"'`)}'`; +} + +function successfulRequiredChecksWithoutE2e() { + return successfulRequiredChecks().filter((check) => check.name !== "E2E / PR Gate"); +} + +function successfulRequiredChecks() { + return REQUIRED_CHECK_NAMES.map((name) => requiredCheck(name)); +} + +function requiredCheck(name: string, conclusion = "SUCCESS") { + const { runId, jobId, workflowName } = REQUIRED_CHECK_RUNS[name]; + return e2eGateCheck([runId, jobId, conclusion, undefined, undefined, workflowName, name]); +} + +function e2eGateCheck(check: E2eCheckFixture, index = 0) { + const [runId, jobId, conclusion, startedAt, detailsUrl, workflowName, name] = check; + return { + __typename: "CheckRun", + name: name ?? "E2E / PR Gate", + workflowName: workflowName ?? "E2E / PR Gate Controller", + detailsUrl: + detailsUrl ?? `https://github.com/NVIDIA/NemoClaw/actions/runs/${runId}/job/${jobId}`, + startedAt: startedAt ?? `2026-01-01T00:${String(index * 2).padStart(2, "0")}:00Z`, + status: "COMPLETED", + conclusion, + }; +} + +function e2eJobs(...ids: number[]): ActionJobFixture[] { + return ids.map((id) => ({ id, name: "E2E / PR Gate" })); +} + +const e2eChecks = (...checks: E2eCheckFixture[]): E2eCheckFixture[] => checks; + +function exactDiffGateRun(result: string, jobs: ActionJobFixture[], attempt = 1): ActionRunFixture { + return { + attempt, + headSha: HEAD_SHA, + headBranch: "feature-branch", + headRepository: "NVIDIA/NemoClaw", + baseSha: BASE_SHA, + displayTitle: `E2E Gate PR #42 head ${HEAD_SHA} base ${BASE_SHA} gate true`, + event: "pull_request_target", + path: ".github/workflows/pr-e2e-gate.yaml", + status: "completed", + conclusion: result, + jobs, + }; +} + +function installerHashRun( + result: string, + jobs: ActionJobFixture[], + gate: boolean, +): ActionRunFixture { + return { + ...exactDiffGateRun(result, jobs), + displayTitle: `Installer Hash PR #42 head ${HEAD_SHA} base ${BASE_SHA} gate ${gate}`, + event: "pull_request", + path: ".github/workflows/installer-hash-check.yaml", + }; +} + +function prWorkflowRun(result: string, jobs: ActionJobFixture[], gate: boolean): ActionRunFixture { + return { + ...exactDiffGateRun(result, jobs), + displayTitle: `CI PR #42 head ${HEAD_SHA} base ${BASE_SHA} gate ${gate}`, + event: "pull_request", + path: ".github/workflows/pr.yaml", + }; +} + +function prWorkflowJobs( + defaultConclusion: string, + overrides: Record> = {}, +): ActionJobFixture[] { + return PR_WORKFLOW_JOB_NAMES.map((name, index) => ({ + id: index + 1, + name, + conclusion: defaultConclusion, + ...overrides[name], + })); +} + +function coordinationCheck(overrides: Record = {}) { + return { + id: 8000, + name: E2E_COORDINATION_NAME, + head_sha: HEAD_SHA, + external_id: E2E_COORDINATION_EXTERNAL_ID, + status: "completed", + conclusion: "success", + started_at: "2026-01-01T00:01:30Z", + completed_at: "2026-01-01T00:02:30Z", + app: { id: 15368 }, + ...overrides, + }; +} + +function e2eRunFixture( + checks: E2eCheckFixture[], + actionRunAttempts: Record, +): ComplianceFixture { + return { + body: "Signed-off-by: Example User ", + verified: true, + statusChecks: [...successfulRequiredChecksWithoutE2e(), ...checks.map(e2eGateCheck)], + actionRunAttempts, + }; +} + +function runGate(fixture: ComplianceFixture) { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "check-gates-compliance-")); + const bin = path.join(tmp, "bin"); + fs.mkdirSync(bin); + const ghPath = path.join(bin, "gh"); + + const pr = { + number: 42, + title: "fix(policy): align maintainer workflow", + url: "https://github.com/NVIDIA/NemoClaw/pull/42", + body: fixture.body, + files: [], + statusCheckRollup: + fixture.statusChecks ?? + (fixture.checkNames ?? REQUIRED_CHECK_NAMES).map((name) => + requiredCheck(name, fixture.checkConclusions?.[name]), + ), + mergeable: fixture.mergeable ?? "MERGEABLE", + mergeStateStatus: fixture.mergeStateStatus ?? "CLEAN", + state: "OPEN", + isDraft: false, + headRefOid: HEAD_SHA, + baseRefOid: BASE_SHA, + headRefName: "feature-branch", + baseRefName: "main", + headRepository: { nameWithOwner: "NVIDIA/NemoClaw" }, + author: { login: fixture.prAuthorLogin ?? "contributor" }, + }; + const finalPr = { ...pr, ...fixture.finalPr }; + const finalPrAfterCurrentBase = { ...finalPr, ...fixture.finalPrAfterCurrentBase }; + const contributorCommitPages = ( + fixture.contributorCommitPages ?? [ + [ + { + authors: (fixture.commitAuthorLogins ?? ["contributor"]).map((login) => ({ + login, + })), + }, + ], + ] + ).map((page) => + page.map((commit) => ({ + ...commit, + authorCount: commit.authorCount ?? commit.authors.length, + })), + ); + const contributorReviewPages = fixture.contributorReviewPages ?? [ + fixture.reviews ?? [ + { + author: { login: "reviewer" }, + state: "APPROVED", + submittedAt: "2026-01-01T00:00:00Z", + }, + ], + ]; + const contributorCommitOutput = contributorCommitPages + .map((page) => + JSON.stringify({ + nodes: page, + totalCount: fixture.contributorCommitTotalCount ?? contributorCommitPages.flat().length, + }), + ) + .join("\n"); + const contributorReviewOutput = contributorReviewPages + .map((page) => + JSON.stringify({ + nodes: page, + totalCount: fixture.contributorReviewTotalCount ?? contributorReviewPages.flat().length, + }), + ) + .join("\n"); + const commit = { + sha: "abc123", + verified: fixture.verified, + reason: fixture.reason ?? (fixture.verified ? "valid" : "unsigned"), + }; + const commitOutput = fixture.commitOutput ?? JSON.stringify(commit); + const currentBaseOutput = JSON.stringify({ + data: { + repository: { + pullRequest: { + baseRef: + fixture.currentBaseSha === null + ? null + : { target: { oid: fixture.currentBaseSha ?? BASE_SHA } }, + }, + }, + }, + }); + const issueEventPages = fixture.issueEventPages ?? [[]]; + const coordinationCheckPages = fixture.coordinationCheckPages ?? [ + { + total_count: 1, + check_runs: [coordinationCheck()], + }, + ]; + const legacyCoordinationCheckPages = fixture.legacyCoordinationCheckPages ?? [ + { total_count: 0, check_runs: [] }, + ]; + const actionRunCases = Object.entries({ + "90": prWorkflowRun( + "success", + [ + { id: 1, name: "checks" }, + { id: 2, name: "changes" }, + ], + true, + ), + "91": { + ...installerHashRun("success", [{ id: 1, name: "check-hash" }], true), + }, + "92": { + ...exactDiffGateRun("success", [{ id: 1, name: "commit-lint" }]), + event: "pull_request", + path: ".github/workflows/commit-lint.yaml", + }, + "93": { + ...exactDiffGateRun("success", [{ id: 1, name: "dco-check" }]), + event: "pull_request", + path: ".github/workflows/dco-check.yaml", + }, + "94": exactDiffGateRun("success", [{ id: 1, name: "E2E / PR Gate" }]), + ...fixture.actionRunAttempts, + }) + .flatMap(([runId, value]) => { + const jobPages = (value.jobPages ?? [value.jobs ?? []]).map((page) => + page.map((job) => ({ + ...job, + status: job.status ?? "completed", + conclusion: job.conclusion === undefined ? "success" : job.conclusion, + })), + ); + const jobs = jobPages.flat(); + const runData = { + run_attempt: value.attempt, + created_at: value.createdAt ?? "2026-01-01T00:01:00Z", + updated_at: value.updatedAt ?? "2026-01-01T00:03:00Z", + event: value.event, + path: value.path, + status: value.status, + conclusion: value.conclusion, + display_title: value.displayTitle, + ...(value.headSha ? { head_sha: value.headSha } : {}), + ...(value.headBranch ? { head_branch: value.headBranch } : {}), + ...(value.headRepository ? { head_repository: { full_name: value.headRepository } } : {}), + ...(value.pullRequests !== undefined + ? { pull_requests: value.pullRequests } + : value.headSha + ? { + pull_requests: value.baseSha + ? [ + { + number: 42, + head: { sha: value.pullRequestHeadSha ?? value.headSha }, + base: { sha: value.baseSha }, + }, + ] + : [], + } + : {}), + }; + const refreshedRunData = { + ...runData, + run_attempt: value.nextAttempt ?? value.attempt, + created_at: value.nextCreatedAt ?? runData.created_at, + updated_at: value.nextUpdatedAt ?? runData.updated_at, + display_title: value.nextDisplayTitle ?? runData.display_title, + status: value.nextStatus ?? runData.status, + conclusion: value.nextConclusion === undefined ? runData.conclusion : value.nextConclusion, + }; + const runMarker = path.join(tmp, `action-run-${runId}-seen`); + return [ + ` "api repos/NVIDIA/NemoClaw/actions/runs/${runId}") if mkdir ${shellSingleQuote(runMarker)} 2>/dev/null; then printf '%s' ${shellSingleQuote(JSON.stringify(runData))}; else printf '%s' ${shellSingleQuote(JSON.stringify(refreshedRunData))}; fi ;;`, + ` "api --paginate --slurp repos/NVIDIA/NemoClaw/actions/runs/${runId}/attempts/${value.attempt}/jobs?per_page=100") printf '%s' ${shellSingleQuote( + JSON.stringify( + jobPages.map((page) => ({ + total_count: jobs.length, + jobs: page, + })), + ), + )} ;;`, + ]; + }) + .join("\n"); + + fs.writeFileSync( + ghPath, + `#!/usr/bin/env bash +set -euo pipefail +case "$*" in + "pr view"*) if mkdir ${shellSingleQuote(path.join(tmp, "pr-view-seen"))} 2>/dev/null; then printf '%s' ${shellSingleQuote(JSON.stringify(pr))}; elif [ -d ${shellSingleQuote(path.join(tmp, "current-base-seen"))} ]; then printf '%s' ${shellSingleQuote(JSON.stringify(finalPrAfterCurrentBase))}; else printf '%s' ${shellSingleQuote(JSON.stringify(finalPr))}; fi ;; + *"ContributorCommits"*) printf '%s' ${shellSingleQuote(contributorCommitOutput)} ;; + *"ContributorReviews"*) printf '%s' ${shellSingleQuote(contributorReviewOutput)} ;; + *"CurrentBaseRef"*) mkdir -p ${shellSingleQuote(path.join(tmp, "current-base-seen"))}; printf '%s' ${shellSingleQuote(currentBaseOutput)} ;; + "api graphql"*) printf '%s' '{"data":{"repository":{"pullRequest":{"reviewThreads":{"nodes":[]}}}}}' ;; + "api repos/NVIDIA/NemoClaw/issues/42/comments"*) printf '%s' '{"id":1,"body":"ordinary comment","user":{"login":"reviewer"},"updated_at":"2026-01-01T00:00:00Z"}' ;; + "api repos/NVIDIA/NemoClaw/pulls/42/commits"*) printf '%s' ${shellSingleQuote(commitOutput)} ;; + "api --paginate --slurp repos/NVIDIA/NemoClaw/issues/42/events?per_page=100") printf '%s' ${shellSingleQuote(JSON.stringify(issueEventPages))} ;; + "api --paginate --slurp repos/NVIDIA/NemoClaw/commits/${HEAD_SHA}/check-runs?check_name=E2E%20%2F%20PR%20Gate%20Coordination&filter=all&per_page=100") printf '%s' ${shellSingleQuote(JSON.stringify(coordinationCheckPages))} ;; + "api --paginate --slurp repos/NVIDIA/NemoClaw/commits/${HEAD_SHA}/check-runs?check_name=E2E%20%2F%20PR%20Gate&filter=all&per_page=100") printf '%s' ${shellSingleQuote(JSON.stringify(legacyCoordinationCheckPages))} ;; +${actionRunCases} + *) echo "unexpected gh args: $*" >&2; exit 9 ;; +esac +`, + ); + fs.chmodSync(ghPath, 0o755); + + try { + return spawnSync( + process.execPath, + [ + "--experimental-strip-types", + "--no-warnings", + ".agents/skills/nemoclaw-maintainer-day/scripts/check-gates.ts", + "42", + ], + { + cwd: process.cwd(), + encoding: "utf-8", + env: { ...process.env, PATH: `${bin}${path.delimiter}${process.env.PATH ?? ""}` }, + }, + ); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } +} + +function runComparatorGate(fixture: ComparatorFixture, prNumber = "42") { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "collect-gates-compliance-")); + const bin = path.join(tmp, "bin"); + fs.mkdirSync(bin); + const ghPath = path.join(bin, "gh"); + + const pr = { + number: Number(prNumber), + state: fixture.state ?? "OPEN", + body: fixture.body, + author: { login: fixture.prAuthorLogin ?? "example-user" }, + headRefOid: fixture.headRefOid ?? "abc123", + statusCheckRollup: (fixture.checkNames ?? REQUIRED_CHECK_NAMES).map((name) => ({ + name, + status: "COMPLETED", + conclusion: fixture.checkConclusions?.[name] ?? "SUCCESS", + })), + mergeable: fixture.mergeable ?? "MERGEABLE", + mergeStateStatus: fixture.mergeStateStatus ?? "CLEAN", + reviewDecision: fixture.reviewDecision ?? "APPROVED", + }; + const commit = { + sha: "abc123", + verified: fixture.verified, + reason: fixture.reason ?? (fixture.verified ? "valid" : "unsigned"), + }; + const commitOutput = fixture.commitOutput ?? JSON.stringify(commit); + + fs.writeFileSync( + ghPath, + `#!/usr/bin/env bash +set -euo pipefail +case "$1 $2" in + "pr view") printf '%s' ${shellSingleQuote(JSON.stringify(pr))} ;; + ${shellSingleQuote(`api repos/NVIDIA/NemoClaw/pulls/${prNumber}/commits`)}) printf '%s' ${shellSingleQuote(commitOutput)} ;; + *) echo "unexpected gh args: $*" >&2; exit 9 ;; +esac +`, + ); + fs.chmodSync(ghPath, 0o755); + + try { + return spawnSync( + "bash", + [ + ".agents/skills/nemoclaw-maintainer-pr-comparator/scripts/collect-gates.sh", + prNumber, + "--repo", + "NVIDIA/NemoClaw", + ], + { + cwd: process.cwd(), + encoding: "utf-8", + env: { ...process.env, PATH: `${bin}${path.delimiter}${process.env.PATH ?? ""}` }, + }, + ); + } finally { + fs.rmSync(tmp, { recursive: true, force: true }); + } +} + +export type { + ActionJobFixture, + ActionRunFixture, + ComparatorFixture, + ComplianceFixture, + E2eCheckFixture, +}; +export { + BASE_SHA, + CUSTOM_RUN_URL, + coordinationCheck, + E2E_COORDINATION_EXTERNAL_ID, + E2E_COORDINATION_NAME, + e2eChecks, + e2eGateCheck, + e2eJobs, + e2eRunFixture, + exactDiffGateRun, + HEAD_SHA, + INCOMPLETE_E2E, + installerHashRun, + prWorkflowJobs, + prWorkflowRun, + REQUIRED_CHECK_NAMES, + requiredCheck, + runComparatorGate, + runGate, + successfulRequiredChecks, + successfulRequiredChecksWithoutE2e, +};