diff --git a/CHANGELOG.md b/CHANGELOG.md index c0d116d58..67d0a1127 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,7 @@ # Changelog ## Unreleased +- GitHub Actions의 queued-without-runner incident RCA를 위해 exact PR source head와 명시적으로 선택한 workflow-run IDs에 결합된 read-only `operations:runner-assignment` audit를 추가했다. Fully paginated `filter=all` job evidence로 runner assignment를 later job conclusion과 분리하며, fresh unassigned queue는 non-passing `PENDING`, bounded grace를 초과한 unassigned job은 fail-closed, runner가 배정된 뒤 workflow가 실패한 경우에는 assignment evidence만 별도로 PASS로 분류한다. Credential-free fixed-path report는 required Check·formal review·merge·release·deployment authority를 대체하지 않는다. - 개발 의존성 체인의 transitive `nanoid` lockfile resolution을 `3.3.16`에서 `3.3.17`로 최소 갱신하여 GHSA-2v37-7h3g-55p8 / CVE-2026-67213 보안 게이트를 복구한다. PostCSS의 선언 범위 `^3.3.16`과 다른 package metadata는 변경하지 않으며 audit waiver·ignore·severity 완화 없이 `npm ci`/`npm audit --audit-level=high`가 exact head에서 재검증되도록 유지한다. - lockfile 재생성 도구 체인을 Node.js 24.19.0/npm 11.17.0으로 정확히 고정하고, `strict-allow-scripts=true` 아래 승인된 install-script identity만 실행하며 schema v3 exact-base lockfile change control로 package metadata drift를 실패-폐쇄한다. exact package before/after digest에 더해 top-level metadata digest와 대규모 package-set bulk evidence를 결합하며, #76의 `nanoid@3.3.17` 보안 수정과 explicit `npm ci --legacy-peer-deps=false --install-links=false` 계약을 보존한다. package-manager/toolchain·install-script authority·vulnerability audit·review/merge authority는 별도 증거 계층으로 유지한다. - `hourly-product-development`가 `NVIDIA_NIM_API_KEY`뿐 아니라 `NOEMA_MAINTAINER_APP_CLIENT_ID`와 `NOEMA_MAINTAINER_APP_PRIVATE_KEY` 존재를 checkout·OpenCode 설치·NVIDIA 호출 전에 검증한다. 게시 경로가 준비되지 않았으면 `maintainer_app_unavailable`로 실패 폐쇄하여 알려진 실패에 추론 비용을 쓰지 않으며, `dry_run`은 credential 없이 queue와 task contract를 검토하는 경로로 유지한다. 기존 reviewer App 및 `NOEMA_LLM_API_KEY`·`contextual-orchestrator` reviewer credential 경계는 변경하지 않는다. diff --git a/docs/doctoring/actions-runner-assignment-audit.md b/docs/doctoring/actions-runner-assignment-audit.md new file mode 100644 index 000000000..3920cc778 --- /dev/null +++ b/docs/doctoring/actions-runner-assignment-audit.md @@ -0,0 +1,102 @@ +# GitHub Actions Runner-Assignment Audit Doctoring + +## Status and purpose + +This record supports issue #30 and the repository command `npm run operations:runner-assignment`. It documents a **read-only operational evidence boundary** for distinguishing a GitHub Actions job that has not yet received a runner from a job that did receive a runner and later succeeded or failed for some other reason. + +This control is deliberately narrower than CI or merge readiness. It does not create a replacement Check, infer source correctness, prove organization billing or runner-policy configuration, authorize a review, or authorize merge/release/deployment. + +## Evidence model + +The operator supplies an explicit bounded set of GitHub Actions workflow-run IDs plus the exact expected pull-request source-head SHA. The collector retrieves each selected workflow run and **all** job pages for that run, then the evaluator classifies runner assignment separately from the later workflow conclusion. + +Required invariants are: + +1. the repository is exactly `ContextualWisdomLab/noema`; +2. `GH_TOKEN` exists only as transport authority for read-only API calls and is never retained in the report; +3. the expected source head is one canonical lowercase 40-character commit SHA; +4. one to twenty unique positive run IDs are selected explicitly; +5. selected runs must be `pull_request` runs bound to that exact source head; +6. job pages are fully paginated with `per_page=100` and `filter=all`, bounded to at most 2,000 retained jobs; +7. runner assignment is observed only from job evidence such as `started_at`, a positive `runner_id`, or a non-empty `runner_name`; +8. a `waiting`, `pending`, or `requested` job remains non-passing `PENDING` because those states do not by themselves isolate runner allocation; +9. a queued job in a run where another job has already received a runner remains non-passing `PENDING`, because the queued job may be waiting on an explicit `jobs..needs` dependency rather than runner capacity; +10. the bounded grace may produce `runner_assignment_stalled` only when the workflow run itself remains `queued`, the job remains `queued`, and no job in that selected run has assignment evidence; +11. an assigned job may produce runner-assignment `PASS` even if its later workflow/test conclusion is `failure`, because those are separate evidence classes. + +The default runner-allocation grace is five minutes and may be bounded by `NOEMA_ACTIONS_AUDIT_QUEUE_GRACE_MILLISECONDS`; the evaluator rejects values above thirty minutes rather than allowing a true isolated queue condition to remain indefinitely pending. + +This classifier is intentionally conservative because the GitHub workflow-job REST representation does not expose a durable repository-consumable timestamp meaning “this job became eligible for runner allocation.” A workflow run's `created_at` is therefore not a trustworthy age for every downstream job. The evaluator uses run age only after the selected evidence isolates the top-level queued runner-allocation boundary described above. + +### Pre-run waits are not runner stalls + +GitHub distinguishes several reasons a job may not yet have reached a runner. In particular, **deployment protection rules** on an environment can leave a deployment job in a waiting state; GitHub documents that a job that references an environment is not sent to a runner until the environment's protection rules pass. Likewise, workflow syntax permits a job to declare `jobs..needs`, so downstream work waits for its prerequisite jobs before it can run. + +Those states remain operationally non-passing, but they are not evidence that GitHub failed to allocate a runner. The audit therefore reports them as `PENDING` without `runner_assignment_stalled`. This does **not** convert them to success: required Checks, approvals, environment protection, and later job conclusions still retain their own authority. + +## Operator contract + +Example: + +```bash +export GH_TOKEN='' +export NOEMA_ACTIONS_AUDIT_REPOSITORY='ContextualWisdomLab/noema' +export NOEMA_ACTIONS_AUDIT_HEAD_SHA='' +export NOEMA_ACTIONS_AUDIT_RUN_IDS='31343034891,31343034896,31343034900' + +npm run operations:runner-assignment +``` + +The command writes the fixed report path: + +```text +artifacts/operations/actions-runner-assignment-audit.json +``` + +The report records repository, expected head, selected run IDs, observation time, queue grace, deterministic checks/failures, and explicit false authority flags for required-check success, review, merge, release, and deployment. Temporary report bytes are created owner-only and atomically renamed onto the fixed report path. + +`PASS` exits zero. `PENDING` and `FAIL` both exit nonzero. A malformed source identity, cross-repository request, missing credential, malformed API JSON, GitHub CLI failure, pagination-shape failure, excessive evidence, or head mismatch fails closed. + +## RCA interpretation + +A `runner_assignment_stalled` result supports the narrow hypothesis **“the selected current-head workflow run and job remained at an isolated queued boundary without observable runner assignment beyond the configured grace interval.”** It does not by itself identify why. Possible causes remain materially distinct and require separate evidence, including GitHub-hosted runner capacity, repository/organization Actions policy, runner-group restrictions, billing/spending controls, concurrency saturation, enterprise policy, or a GitHub service incident. + +A `PENDING` result for environment protection, `needs` dependency waiting, or other pre-run uncertainty means only that runner allocation has **not been isolated as the failing boundary**. It is not a health PASS and cannot satisfy a required Check. + +Conversely, an observed runner assignment falsifies the hypothesis that the specific selected job is still blocked at runner allocation. A later failing step must be investigated at that later boundary rather than described as a runner-assignment incident. + +This separation matters for issue #30 because historical Noema runs exhibited queued jobs without logs, while later runs demonstrably received GitHub-hosted runners. Repository evidence therefore needs to preserve **assignment state** independently from **job conclusion**, **dependency/protection waiting**, and any organization-level causal claim. + +## Security and privacy + +The collector uses GitHub Actions REST **read** endpoints only. It does not rerun, cancel, dispatch, approve, merge, modify refs, or change settings. The report does not contain `GH_TOKEN`, repository secrets, workflow logs, source contents, personal data beyond ordinary GitHub workflow/job metadata needed for the operational decision, or model output. + +The `gh` child process also receives a purpose-built minimal environment rather than ambient process state: only `PATH`, the read-only `GH_TOKEN`, pinned `GH_HOST=github.com`, and `NO_COLOR=1` cross the process boundary. `GITHUB_TOKEN`, `NVIDIA_NIM_API_KEY`, Maintainer/Reviewer App private material, `HOME`, and ambient proxy variables are excluded. This prevents a read-only diagnostic subprocess from accidentally inheriting stronger publication/model credentials or redirecting credential-bearing requests through an unreviewed proxy path. + +The audit is diagnostic evidence. A passing assignment audit cannot satisfy branch protection, required checks, formal review, security scanning, release provenance, production deployment, or acquisition evidence. + +## Acceptance + +The repository-owned slice is acceptable when: + +- realistic tests reproduce an isolated runner stall, a fresh queue, deployment/environment waiting, downstream dependency waiting, assigned-but-failed jobs, head mismatch, malformed evidence, pagination, and bounded selection; +- environment-protected and dependency-blocked jobs remain nonzero `PENDING` and are not mislabeled as runner-allocation stalls; +- the pure evaluator and bounded source collector are GREEN; +- the operator adapter performs only the two documented read families and fully paginates jobs; +- the `gh` subprocess inherits only the minimal read-authority environment documented above; +- `PENDING` remains nonzero; +- report output is credential-free and authority-separated; +- normal repository tests and configured production coverage remain intact; +- the active PR does not claim issue #30 operational closure merely because the diagnostic control exists. + +Closing issue #30 still requires real live evidence for the repository/organization acceptance criteria recorded in that issue. + +## References + +GitHub. (2026). *REST API endpoints for workflow jobs*. GitHub Docs. https://docs.github.com/en/rest/actions/workflow-jobs?apiVersion=2026-03-10 + +GitHub. (2026). *REST API endpoints for workflow runs*. GitHub Docs. https://docs.github.com/en/rest/actions/workflow-runs?apiVersion=2026-03-10 + +GitHub. (2026). *Deployments and environments*. GitHub Docs. https://docs.github.com/en/actions/concepts/workflows-and-actions/deployment-environments + +GitHub. (2026). *Workflow syntax for GitHub Actions*. GitHub Docs. https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#jobsjob_idneeds diff --git a/package.json b/package.json index 1959a0948..e909454bd 100644 --- a/package.json +++ b/package.json @@ -45,6 +45,7 @@ "typecheck": "tsc --noEmit", "governance:audit": "node scripts/main-governance-audit.mjs", "operations:preflight": "node scripts/maintainer-app-readiness.mjs", + "operations:runner-assignment": "node scripts/actions-runner-assignment-audit.mjs", "production:governance": "node scripts/production-environment-governance-audit.mjs", "production:preflight": "node scripts/production-evidence-preflight.mjs", "acquisition:deployment-evidence": "node scripts/acquisition-deployment-evidence-audit.mjs", diff --git a/scripts/actions-runner-assignment-audit.mjs b/scripts/actions-runner-assignment-audit.mjs new file mode 100644 index 000000000..e09422a7a --- /dev/null +++ b/scripts/actions-runner-assignment-audit.mjs @@ -0,0 +1,282 @@ +#!/usr/bin/env node + +import { + closeSync, + mkdirSync, + openSync, + renameSync, + unlinkSync, + writeFileSync, +} from "node:fs"; +import { dirname, resolve } from "node:path"; +import { pathToFileURL } from "node:url"; +import { randomUUID } from "node:crypto"; +import { spawnSync } from "node:child_process"; +import { + DEFAULT_RUNNER_QUEUE_GRACE_MILLISECONDS, + evaluateRunnerAssignmentEvidence, +} from "./lib/actions-runner-assignment-audit.mjs"; +import { + collectRunnerAssignmentEvidence, + parseSelectedRunIds, +} from "./lib/actions-runner-assignment-source.mjs"; + +const AUDITED_REPOSITORY = "ContextualWisdomLab/noema"; +const GITHUB_API_VERSION = "2026-03-10"; +const GH_API_TIMEOUT_MILLISECONDS = 20_000; +const GH_API_MAX_BUFFER_BYTES = 2 * 1024 * 1024; +const REPORT_PATH = "artifacts/operations/actions-runner-assignment-audit.json"; +const canonicalShaPattern = /^[0-9a-f]{40}$/; + +function boundedErrorText(value) { + const text = typeof value === "string" ? value : String(value ?? ""); + return text.replace(/[\u0000-\u001f\u007f]/g, " ").trim().slice(0, 1000); +} + +/** + * Build the least-privilege environment inherited by the `gh` subprocess. + * + * Only executable lookup and the read-only GitHub CLI authentication contract + * cross the process boundary. Repository-write tokens, model credentials, + * Maintainer/Reviewer App secrets, proxy settings, HOME-scoped credentials, + * and other ambient process state are deliberately excluded. + * + * @param {unknown} environment Untrusted parent-process environment mapping. + * @returns {{PATH: string, GH_TOKEN: string, GH_HOST: string, NO_COLOR: string}} + * Minimal GitHub CLI environment pinned to GitHub Cloud. + */ +export function createGhSubprocessEnvironment(environment) { + if (!environment || typeof environment !== "object") { + throw new Error("GitHub CLI subprocess environment must be an object."); + } + const executablePath = environment.PATH; + if (typeof executablePath !== "string" || executablePath.trim().length === 0) { + throw new Error("PATH is required for the GitHub CLI subprocess."); + } + const token = environment.GH_TOKEN; + if (typeof token !== "string" || token.trim().length === 0) { + throw new Error("GH_TOKEN is required for the GitHub CLI subprocess."); + } + return { + PATH: executablePath, + GH_TOKEN: token, + GH_HOST: "github.com", + NO_COLOR: "1", + }; +} + +/** + * Read one GitHub REST resource through the authenticated `gh` CLI. + * + * The caller supplies only repository-relative API paths. Pagination uses + * `--slurp` so every returned page remains explicit to the bounded source + * collector instead of being silently collapsed or truncated. + */ +export function ghApi(path, options = {}) { + if (typeof path !== "string" || path.length === 0 || path.length > 1000) { + throw new Error("GitHub API path is invalid."); + } + if (path.startsWith("/") || path.includes("..") || /[\u0000-\u001f\u007f]/.test(path)) { + throw new Error("GitHub API path is outside the bounded relative-path contract."); + } + + const args = [ + "api", + "-H", + "Accept: application/vnd.github+json", + "-H", + `X-GitHub-Api-Version: ${GITHUB_API_VERSION}`, + ]; + if (options.paginate === true) { + args.push("--paginate", "--slurp"); + } + args.push(path); + + const result = spawnSync("gh", args, { + encoding: "utf8", + timeout: GH_API_TIMEOUT_MILLISECONDS, + maxBuffer: GH_API_MAX_BUFFER_BYTES, + env: createGhSubprocessEnvironment(process.env), + stdio: ["ignore", "pipe", "pipe"], + }); + + if (result.error) { + throw new Error(`GitHub Actions evidence read failed: ${boundedErrorText(result.error.message)}`); + } + if (result.status !== 0) { + throw new Error( + `GitHub Actions evidence read failed with gh exit ${result.status}: ${boundedErrorText(result.stderr)}`, + ); + } + + try { + return JSON.parse(result.stdout); + } catch { + throw new Error("GitHub Actions evidence read returned malformed JSON."); + } +} + +/** + * Create read-only GitHub Actions REST adapters for the operator audit. + * + * @param {{repository: string, gh_api: Function}} input Repository and API reader. + * @returns {{fetch_run: Function, fetch_job_pages: Function}} Bounded read adapters. + */ +export function createGhReadAdapters(input) { + if (!input || input.repository !== AUDITED_REPOSITORY) { + throw new Error(`Runner-assignment audit is restricted to ${AUDITED_REPOSITORY}.`); + } + if (typeof input.gh_api !== "function") { + throw new Error("A read-only GitHub API adapter is required."); + } + + return { + fetch_run: async (runId) => + input.gh_api(`repos/${AUDITED_REPOSITORY}/actions/runs/${runId}`, { + paginate: false, + }), + fetch_job_pages: async (runId) => + input.gh_api( + `repos/${AUDITED_REPOSITORY}/actions/runs/${runId}/jobs?filter=all&per_page=100`, + { paginate: true }, + ), + }; +} + +function parseQueueGrace(value) { + if (value === undefined || value === "") { + return DEFAULT_RUNNER_QUEUE_GRACE_MILLISECONDS; + } + if (typeof value !== "string" || !/^[1-9][0-9]*$/.test(value)) { + throw new Error("NOEMA_ACTIONS_AUDIT_QUEUE_GRACE_MILLISECONDS must be a positive integer."); + } + const parsed = Number(value); + if (!Number.isSafeInteger(parsed)) { + throw new Error("NOEMA_ACTIONS_AUDIT_QUEUE_GRACE_MILLISECONDS must be a safe integer."); + } + return parsed; +} + +/** Write the fixed audit report atomically with owner-only temporary permissions. */ +export function writeReportAtomically(report) { + const reportPath = resolve(REPORT_PATH); + const reportDirectory = dirname(reportPath); + mkdirSync(reportDirectory, { recursive: true, mode: 0o700 }); + const temporaryPath = `${reportPath}.tmp-${process.pid}-${randomUUID()}`; + let descriptor; + try { + descriptor = openSync(temporaryPath, "wx", 0o600); + writeFileSync(descriptor, `${JSON.stringify(report, null, 2)}\n`, "utf8"); + closeSync(descriptor); + descriptor = undefined; + renameSync(temporaryPath, reportPath); + } finally { + if (descriptor !== undefined) { + closeSync(descriptor); + } + try { + unlinkSync(temporaryPath); + } catch (error) { + if (error?.code !== "ENOENT") { + throw error; + } + } + } +} + +/** + * Execute the runner-assignment audit from explicit operator inputs. + * + * `PENDING` is deliberately a nonzero result. A runner being assigned is also + * not a substitute for the later workflow/check conclusion, review, merge, + * release, or deployment authority. + */ +export async function runActionsRunnerAssignmentAudit(input) { + if (!input || typeof input !== "object") { + throw new Error("Runner-assignment audit input must be an object."); + } + const env = input.env; + if (!env || typeof env !== "object") { + throw new Error("Runner-assignment audit environment is required."); + } + if (typeof env.GH_TOKEN !== "string" || env.GH_TOKEN.trim().length === 0) { + throw new Error("GH_TOKEN is required for read-only GitHub Actions evidence collection."); + } + + const repository = env.NOEMA_ACTIONS_AUDIT_REPOSITORY; + if (repository !== AUDITED_REPOSITORY) { + throw new Error(`NOEMA_ACTIONS_AUDIT_REPOSITORY must equal ${AUDITED_REPOSITORY}.`); + } + + const expectedHeadSha = env.NOEMA_ACTIONS_AUDIT_HEAD_SHA; + if (typeof expectedHeadSha !== "string" || !canonicalShaPattern.test(expectedHeadSha)) { + throw new Error("NOEMA_ACTIONS_AUDIT_HEAD_SHA must be a canonical lowercase 40-character SHA."); + } + const runIds = parseSelectedRunIds(env.NOEMA_ACTIONS_AUDIT_RUN_IDS); + const queueGrace = parseQueueGrace(env.NOEMA_ACTIONS_AUDIT_QUEUE_GRACE_MILLISECONDS); + if (typeof input.observed_at !== "string" || !Number.isFinite(Date.parse(input.observed_at))) { + throw new Error("observed_at must be a parseable timestamp."); + } + if (typeof input.write_report !== "function") { + throw new Error("A report writer is required."); + } + + const adapters = createGhReadAdapters({ + repository, + gh_api: input.gh_api, + }); + const evidence = await collectRunnerAssignmentEvidence({ + expected_head_sha: expectedHeadSha, + observed_at: input.observed_at, + queue_grace_milliseconds: queueGrace, + run_ids: runIds, + fetch_run: adapters.fetch_run, + fetch_job_pages: adapters.fetch_job_pages, + }); + const decision = evaluateRunnerAssignmentEvidence(evidence); + const report = { + schema_version: 1, + objective: "github_actions_runner_assignment", + repository, + expected_head_sha: expectedHeadSha, + selected_run_ids: runIds, + observed_at: input.observed_at, + queue_grace_milliseconds: queueGrace, + status: decision.status, + checks: decision.checks, + failures: decision.failures, + authority: { + runner_assignment_only: true, + required_check_success: false, + review_authority: false, + merge_authority: false, + release_authority: false, + deployment_authority: false, + }, + }; + await input.write_report(report); + + return { + exit_code: decision.status === "PASS" ? 0 : 1, + report, + }; +} + +async function main() { + const result = await runActionsRunnerAssignmentAudit({ + env: process.env, + observed_at: new Date().toISOString(), + gh_api: ghApi, + write_report: writeReportAtomically, + }); + process.stdout.write(`${result.report.status}\n`); + process.exitCode = result.exit_code; +} + +const invokedPath = process.argv[1] ? pathToFileURL(resolve(process.argv[1])).href : ""; +if (invokedPath === import.meta.url) { + main().catch((error) => { + process.stderr.write(`runner-assignment audit failed: ${boundedErrorText(error?.message)}\n`); + process.exitCode = 2; + }); +} diff --git a/scripts/lib/actions-runner-assignment-audit.mjs b/scripts/lib/actions-runner-assignment-audit.mjs new file mode 100644 index 000000000..f7dba4242 --- /dev/null +++ b/scripts/lib/actions-runner-assignment-audit.mjs @@ -0,0 +1,266 @@ +export const DEFAULT_RUNNER_QUEUE_GRACE_MILLISECONDS = 5 * 60 * 1000; +const MAX_RUNNER_QUEUE_GRACE_MILLISECONDS = 30 * 60 * 1000; +const canonicalShaPattern = /^[0-9a-f]{40}$/; +const pendingJobStatuses = new Set(["queued", "requested", "waiting", "pending"]); + +function failure(code, detail, context = {}) { + return { code, detail, ...context }; +} + +function check(code, pass, detail, context = {}) { + return { code, pass, detail, ...context }; +} + +function parseTimestamp(value) { + if (typeof value !== "string" || value.length === 0 || value.length > 100) { + return null; + } + const milliseconds = Date.parse(value); + return Number.isFinite(milliseconds) ? milliseconds : null; +} + +function positiveSafeInteger(value) { + return Number.isSafeInteger(value) && value > 0; +} + +function boundedName(value) { + if (typeof value !== "string") { + return "unknown"; + } + const text = value.replace(/[\u0000-\u001f\u007f]/g, "").trim(); + return text.length === 0 ? "unknown" : text.slice(0, 300); +} + +function assignmentObserved(job) { + const startedAt = parseTimestamp(job?.started_at); + const runnerId = job?.runner_id; + const runnerName = typeof job?.runner_name === "string" ? job.runner_name.trim() : ""; + return startedAt !== null || positiveSafeInteger(runnerId) || runnerName.length > 0; +} + +function invalidEvidence(detail) { + return { + status: "FAIL", + checks: [], + failures: [failure("runner_evidence_invalid", detail)], + }; +} + +/** + * Evaluate bounded GitHub Actions runner-assignment evidence. + * + * The evaluator answers only whether each selected pull-request workflow job + * obtained a runner. A later test, security, or workflow conclusion remains a + * separate evidence class. Freshly queued jobs remain non-passing `PENDING`. + * A grace-window stall is emitted only when both the workflow run and the job + * remain queued with no assignment observed anywhere in the selected run; + * protection/dependency waits stay non-passing without being mislabeled as a + * runner-allocation failure. + * + * @param {unknown} evidence Untrusted workflow-run and job evidence. + * @returns {{status: "PASS" | "PENDING" | "FAIL", checks: object[], failures: object[]}} + * A deterministic assignment decision that never substitutes for a required + * GitHub Check, review, merge, release, or deployment authority. + */ +export function evaluateRunnerAssignmentEvidence(evidence) { + if (!evidence || typeof evidence !== "object" || !Array.isArray(evidence.runs)) { + return invalidEvidence("Runner-assignment evidence must contain a runs array."); + } + + const expectedHeadSha = evidence.expected_head_sha; + if (typeof expectedHeadSha !== "string" || !canonicalShaPattern.test(expectedHeadSha)) { + return invalidEvidence("expected_head_sha must be a canonical 40-character lowercase commit SHA."); + } + + const observedAt = parseTimestamp(evidence.observed_at); + if (observedAt === null) { + return invalidEvidence("observed_at must be a parseable timestamp."); + } + + const queueGrace = evidence.queue_grace_milliseconds + ?? DEFAULT_RUNNER_QUEUE_GRACE_MILLISECONDS; + if ( + !Number.isSafeInteger(queueGrace) + || queueGrace <= 0 + || queueGrace > MAX_RUNNER_QUEUE_GRACE_MILLISECONDS + ) { + return invalidEvidence( + `queue_grace_milliseconds must be an integer between 1 and ${MAX_RUNNER_QUEUE_GRACE_MILLISECONDS}.`, + ); + } + + if (evidence.runs.length === 0) { + return { + status: "FAIL", + checks: [], + failures: [ + failure( + "workflow_run_evidence_missing", + "At least one current-head workflow run is required for runner-assignment evidence.", + ), + ], + }; + } + + const checks = []; + const failures = []; + let pending = false; + + for (const run of evidence.runs) { + if (!run || typeof run !== "object" || !positiveSafeInteger(run.id)) { + failures.push( + failure("workflow_run_invalid", "Each workflow run must include a positive integer id."), + ); + continue; + } + + const runContext = { + run_id: run.id, + workflow_name: boundedName(run.name), + }; + + if (run.head_sha !== expectedHeadSha) { + failures.push( + failure( + "workflow_run_head_mismatch", + "Workflow-run evidence is not bound to the expected pull-request source head.", + runContext, + ), + ); + continue; + } + + if (run.event !== "pull_request") { + failures.push( + failure( + "workflow_run_event_invalid", + "Runner-assignment evidence must come from a pull_request workflow run.", + runContext, + ), + ); + continue; + } + + const createdAt = parseTimestamp(run.created_at); + if (createdAt === null || createdAt > observedAt) { + failures.push( + failure( + "workflow_run_timestamp_invalid", + "Workflow-run created_at must be a parseable timestamp no later than observed_at.", + runContext, + ), + ); + continue; + } + + if (!Array.isArray(run.jobs) || run.jobs.length === 0) { + failures.push( + failure( + "workflow_job_evidence_missing", + "Each selected workflow run must include at least one job record.", + runContext, + ), + ); + continue; + } + + const runStatus = boundedName(run.status).toLowerCase(); + const runHasAssignment = run.jobs.some(assignmentObserved); + + for (const job of run.jobs) { + if (!job || typeof job !== "object" || !positiveSafeInteger(job.id)) { + failures.push( + failure( + "workflow_job_invalid", + "Each workflow job must include a positive integer id.", + runContext, + ), + ); + continue; + } + + const jobContext = { + ...runContext, + job_id: job.id, + job_name: boundedName(job.name), + }; + + if (assignmentObserved(job)) { + checks.push( + check( + "runner_assignment_observed", + true, + "GitHub job evidence shows that a runner was assigned; the later job conclusion remains separate.", + jobContext, + ), + ); + continue; + } + + const jobStatus = boundedName(job.status).toLowerCase(); + if (!pendingJobStatuses.has(jobStatus)) { + failures.push( + failure( + "runner_assignment_not_observed", + "The job reached a non-queue state without trustworthy started_at or runner identity evidence.", + { ...jobContext, job_status: jobStatus }, + ), + ); + continue; + } + + const runnerQueueIsIsolated = jobStatus === "queued" + && runStatus === "queued" + && !runHasAssignment; + + if (!runnerQueueIsIsolated) { + pending = true; + checks.push( + check( + "runner_assignment_pending", + false, + "The job is non-passing, but current evidence does not isolate runner allocation from dependency or protection-rule waiting.", + { ...jobContext, job_status: jobStatus, run_status: runStatus }, + ), + ); + continue; + } + + const queuedMilliseconds = observedAt - createdAt; + if (queuedMilliseconds > queueGrace) { + failures.push( + failure( + "runner_assignment_stalled", + "The current-head run and job remained queued without runner-assignment evidence beyond the bounded grace window.", + { ...jobContext, queued_milliseconds: queuedMilliseconds }, + ), + ); + checks.push( + check( + "runner_assignment_stalled", + false, + "Runner assignment was not observed before the bounded queue grace elapsed after the queue boundary was isolated.", + { ...jobContext, queued_milliseconds: queuedMilliseconds }, + ), + ); + continue; + } + + pending = true; + checks.push( + check( + "runner_assignment_pending", + false, + "The current-head run and job remain queued inside the bounded runner-assignment grace and are not passing evidence.", + { ...jobContext, queued_milliseconds: queuedMilliseconds }, + ), + ); + } + } + + return { + status: failures.length > 0 ? "FAIL" : pending ? "PENDING" : "PASS", + checks, + failures, + }; +} diff --git a/scripts/lib/actions-runner-assignment-source.mjs b/scripts/lib/actions-runner-assignment-source.mjs new file mode 100644 index 000000000..0cad167a3 --- /dev/null +++ b/scripts/lib/actions-runner-assignment-source.mjs @@ -0,0 +1,146 @@ +const MAX_SELECTED_RUNS = 20; +const MAX_SELECTED_JOBS = 2000; +const MAX_RUN_ID_TEXT_BYTES = 1000; + +function positiveSafeInteger(value) { + return Number.isSafeInteger(value) && value > 0; +} + +/** + * Parse a bounded comma-separated GitHub Actions run-id selection. + * + * @param {unknown} value Operator-supplied comma-separated numeric run IDs. + * @returns {number[]} Unique positive run IDs in the supplied order. + */ +export function parseSelectedRunIds(value) { + if (typeof value !== "string" || value.trim().length === 0) { + throw new Error("Select at least one GitHub Actions run id."); + } + if (Buffer.byteLength(value, "utf8") > MAX_RUN_ID_TEXT_BYTES) { + throw new Error(`Run-id selection must be at most ${MAX_RUN_ID_TEXT_BYTES} bytes.`); + } + + const tokens = value.split(",").map((token) => token.trim()); + if (tokens.length > MAX_SELECTED_RUNS) { + throw new Error(`Select at most ${MAX_SELECTED_RUNS} workflow runs per audit.`); + } + + const runIds = []; + const seen = new Set(); + for (const token of tokens) { + if (!/^[1-9][0-9]*$/.test(token)) { + throw new Error("Every selected GitHub Actions run id must be a positive integer."); + } + const runId = Number(token); + if (!positiveSafeInteger(runId)) { + throw new Error("Every selected GitHub Actions run id must be a positive integer."); + } + if (seen.has(runId)) { + throw new Error("Selected GitHub Actions run ids must be unique."); + } + seen.add(runId); + runIds.push(runId); + } + + return runIds; +} + +/** + * Flatten paginated GitHub Actions job pages without trusting page shape. + * + * @param {unknown} pages Slurped `gh api --paginate` page objects. + * @returns {object[]} A bounded list containing every job from every page. + */ +export function flattenJobPages(pages) { + if (!Array.isArray(pages)) { + throw new Error("Workflow job pages must be supplied as an array."); + } + + const jobs = []; + for (const page of pages) { + if (!page || typeof page !== "object" || !Array.isArray(page.jobs)) { + throw new Error("Each workflow job page must contain a jobs array."); + } + if (jobs.length + page.jobs.length > MAX_SELECTED_JOBS) { + throw new Error(`Workflow job evidence exceeds the ${MAX_SELECTED_JOBS}-job bound.`); + } + jobs.push(...page.jobs); + } + return jobs; +} + +function projectRun(run) { + if (!run || typeof run !== "object") { + throw new Error("GitHub workflow-run evidence must be an object."); + } + return { + id: run.id, + name: run.name, + event: run.event, + head_sha: run.head_sha, + status: run.status, + conclusion: run.conclusion, + created_at: run.created_at, + }; +} + +function projectJob(job) { + if (!job || typeof job !== "object") { + throw new Error("GitHub workflow-job evidence must be an object."); + } + return { + id: job.id, + name: job.name, + status: job.status, + conclusion: job.conclusion, + started_at: job.started_at, + completed_at: job.completed_at, + runner_id: job.runner_id, + runner_name: job.runner_name, + }; +} + +/** + * Collect exact-head workflow-run/job evidence through injected read-only adapters. + * + * Network transport is deliberately outside this function. Callers provide one + * read adapter for a workflow run and one for its fully paginated job pages; + * only the bounded fields consumed by runner-assignment evaluation are retained. + * + * @param {object} input Source identity, selected runs, and read adapters. + * @returns {Promise} Evidence ready for deterministic assignment evaluation. + */ +export async function collectRunnerAssignmentEvidence(input) { + if (!input || typeof input !== "object") { + throw new Error("Runner-assignment source input must be an object."); + } + if (!Array.isArray(input.run_ids) || input.run_ids.length === 0) { + throw new Error("At least one selected workflow run is required."); + } + if (input.run_ids.length > MAX_SELECTED_RUNS) { + throw new Error(`Select at most ${MAX_SELECTED_RUNS} workflow runs per audit.`); + } + if (!input.run_ids.every(positiveSafeInteger) || new Set(input.run_ids).size !== input.run_ids.length) { + throw new Error("Selected workflow run ids must be unique positive integers."); + } + if (typeof input.fetch_run !== "function" || typeof input.fetch_job_pages !== "function") { + throw new Error("Read-only workflow-run and job-page adapters are required."); + } + + const runs = []; + for (const runId of input.run_ids) { + const run = projectRun(await input.fetch_run(runId)); + const jobPages = await input.fetch_job_pages(runId); + const jobs = flattenJobPages(jobPages).map(projectJob); + runs.push({ ...run, jobs }); + } + + return { + expected_head_sha: input.expected_head_sha, + observed_at: input.observed_at, + queue_grace_milliseconds: input.queue_grace_milliseconds, + runs, + }; +} + +export { MAX_SELECTED_RUNS }; diff --git a/test/actions-runner-assignment-audit.test.ts b/test/actions-runner-assignment-audit.test.ts new file mode 100644 index 000000000..45ca88e2a --- /dev/null +++ b/test/actions-runner-assignment-audit.test.ts @@ -0,0 +1,143 @@ +import { describe, expect, it } from "vitest"; +import { + DEFAULT_RUNNER_QUEUE_GRACE_MILLISECONDS, + evaluateRunnerAssignmentEvidence, +} from "../scripts/lib/actions-runner-assignment-audit.mjs"; + +const expectedHead = "0123456789abcdef0123456789abcdef01234567"; +const observedAt = "2026-08-10T00:00:00.000Z"; + +function workflowRun(overrides: Record = {}) { + return { + id: 101, + name: "ci", + event: "pull_request", + head_sha: expectedHead, + status: "queued", + conclusion: null, + created_at: "2026-08-09T23:50:00.000Z", + jobs: [ + { + id: 201, + name: "verify", + status: "queued", + conclusion: null, + started_at: null, + completed_at: null, + runner_id: null, + runner_name: null, + }, + ], + ...overrides, + }; +} + +function evaluate(runs: unknown[]) { + return evaluateRunnerAssignmentEvidence({ + expected_head_sha: expectedHead, + observed_at: observedAt, + queue_grace_milliseconds: DEFAULT_RUNNER_QUEUE_GRACE_MILLISECONDS, + runs, + }); +} + +function failureCodes(result: ReturnType) { + return result.failures.map((failure: { code?: string }) => failure.code); +} + +describe("GitHub Actions runner-assignment evidence", () => { + it("fails closed when a current-head job remains unassigned beyond the grace window", () => { + const result = evaluate([workflowRun()]); + expect(result.status).toBe("FAIL"); + expect(failureCodes(result)).toContain("runner_assignment_stalled"); + }); + + it("keeps a recently queued unassigned job pending rather than calling it healthy", () => { + const result = evaluate([workflowRun({ created_at: "2026-08-09T23:58:00.000Z" })]); + expect(result.status).toBe("PENDING"); + expect(result.checks).toContainEqual(expect.objectContaining({ code: "runner_assignment_pending", pass: false })); + }); + + it("does not call an environment-protected waiting job a runner-assignment stall", () => { + const result = evaluate([workflowRun({ + status: "waiting", + jobs: [{ + id: 201, + name: "deploy", + status: "waiting", + conclusion: null, + started_at: null, + completed_at: null, + runner_id: null, + runner_name: null, + }], + })]); + expect(result.status).toBe("PENDING"); + expect(failureCodes(result)).not.toContain("runner_assignment_stalled"); + expect(result.checks).toContainEqual(expect.objectContaining({ code: "runner_assignment_pending", pass: false })); + }); + + it("does not age a downstream queued job from workflow creation after another job has started", () => { + const result = evaluate([workflowRun({ + status: "in_progress", + jobs: [ + { + id: 201, + name: "build", + status: "in_progress", + conclusion: null, + started_at: "2026-08-09T23:51:00.000Z", + completed_at: null, + runner_id: 77, + runner_name: "GitHub Actions 77", + }, + { + id: 202, + name: "package", + status: "queued", + conclusion: null, + started_at: null, + completed_at: null, + runner_id: null, + runner_name: null, + }, + ], + })]); + expect(result.status).toBe("PENDING"); + expect(failureCodes(result)).not.toContain("runner_assignment_stalled"); + expect(result.checks).toContainEqual(expect.objectContaining({ code: "runner_assignment_observed", pass: true, job_id: 201 })); + expect(result.checks).toContainEqual(expect.objectContaining({ code: "runner_assignment_pending", pass: false, job_id: 202 })); + }); + + it("proves runner assignment independently from the later job conclusion", () => { + const result = evaluate([workflowRun({ + status: "completed", + conclusion: "failure", + jobs: [{ + id: 201, + name: "verify", + status: "completed", + conclusion: "failure", + started_at: "2026-08-09T23:52:00.000Z", + completed_at: "2026-08-09T23:53:00.000Z", + runner_id: 77, + runner_name: "GitHub Actions 77", + }], + })]); + expect(result.status).toBe("PASS"); + expect(result.failures).toEqual([]); + expect(result.checks).toContainEqual(expect.objectContaining({ code: "runner_assignment_observed", pass: true })); + }); + + it("rejects workflow evidence from a different source head", () => { + const result = evaluate([workflowRun({ head_sha: "fedcba9876543210fedcba9876543210fedcba98" })]); + expect(result.status).toBe("FAIL"); + expect(failureCodes(result)).toContain("workflow_run_head_mismatch"); + }); + + it("fails closed when no workflow-run evidence is supplied", () => { + const result = evaluate([]); + expect(result.status).toBe("FAIL"); + expect(failureCodes(result)).toContain("workflow_run_evidence_missing"); + }); +}); diff --git a/test/actions-runner-assignment-cli.test.ts b/test/actions-runner-assignment-cli.test.ts new file mode 100644 index 000000000..93e24358f --- /dev/null +++ b/test/actions-runner-assignment-cli.test.ts @@ -0,0 +1,89 @@ +import { describe, expect, it, vi } from "vitest"; +import { + createGhReadAdapters, + createGhSubprocessEnvironment, + runActionsRunnerAssignmentAudit, +} from "../scripts/actions-runner-assignment-audit.mjs"; + +const expectedHead = "0123456789abcdef0123456789abcdef01234567"; + +describe("runner-assignment operator audit", () => { + it("uses only bounded read-only workflow-run and fully paginated job endpoints", async () => { + const ghApi = vi.fn(async (path: string) => { + if (path.endsWith("/jobs?filter=all&per_page=100")) { + return [{ jobs: [{ id: 1001 }] }, { jobs: [{ id: 1002 }] }]; + } + return { id: 100, head_sha: expectedHead, event: "pull_request" }; + }); + const adapters = createGhReadAdapters({ repository: "ContextualWisdomLab/noema", gh_api: ghApi }); + await expect(adapters.fetch_run(100)).resolves.toMatchObject({ id: 100 }); + await expect(adapters.fetch_job_pages(100)).resolves.toEqual([{ jobs: [{ id: 1001 }] }, { jobs: [{ id: 1002 }] }]); + expect(ghApi).toHaveBeenNthCalledWith(1, "repos/ContextualWisdomLab/noema/actions/runs/100", { paginate: false }); + expect(ghApi).toHaveBeenNthCalledWith(2, "repos/ContextualWisdomLab/noema/actions/runs/100/jobs?filter=all&per_page=100", { paginate: true }); + }); + + it("isolates the gh subprocess from unrelated repository, model, and proxy credentials", () => { + expect(createGhSubprocessEnvironment({ + PATH: "/usr/bin:/bin", + GH_TOKEN: "read-only-token", + GITHUB_TOKEN: "must-not-cross", + NVIDIA_NIM_API_KEY: "must-not-cross", + NOEMA_MAINTAINER_APP_PRIVATE_KEY: "must-not-cross", + HTTPS_PROXY: "https://ambient-proxy.invalid", + HOME: "/home/runner", + })).toEqual({ + PATH: "/usr/bin:/bin", + GH_TOKEN: "read-only-token", + GH_HOST: "github.com", + NO_COLOR: "1", + }); + }); + + it("fails closed before spawning gh when the minimal subprocess environment is incomplete", () => { + expect(() => createGhSubprocessEnvironment(null)).toThrow("environment"); + expect(() => createGhSubprocessEnvironment({ GH_TOKEN: "token" })).toThrow("PATH"); + expect(() => createGhSubprocessEnvironment({ PATH: "/usr/bin" })).toThrow("GH_TOKEN"); + }); + + it("returns nonzero for fresh pending assignment without promoting it to success", async () => { + const writeReport = vi.fn(); + const ghApi = vi.fn(async (path: string) => { + if (path.endsWith("/jobs?filter=all&per_page=100")) { + return [{ jobs: [{ id: 1001, name: "verify", status: "queued", conclusion: null, started_at: null, completed_at: null, runner_id: null, runner_name: null }] }]; + } + return { id: 100, name: "ci", event: "pull_request", head_sha: expectedHead, status: "queued", conclusion: null, created_at: "2026-08-09T23:58:00.000Z" }; + }); + const result = await runActionsRunnerAssignmentAudit({ + env: { + GH_TOKEN: "present-but-never-retained", + NOEMA_ACTIONS_AUDIT_REPOSITORY: "ContextualWisdomLab/noema", + NOEMA_ACTIONS_AUDIT_HEAD_SHA: expectedHead, + NOEMA_ACTIONS_AUDIT_RUN_IDS: "100", + }, + observed_at: "2026-08-10T00:00:00.000Z", + gh_api: ghApi, + write_report: writeReport, + }); + expect(result.exit_code).toBe(1); + expect(result.report).toMatchObject({ schema_version: 1, objective: "github_actions_runner_assignment", repository: "ContextualWisdomLab/noema", expected_head_sha: expectedHead, selected_run_ids: [100], status: "PENDING" }); + expect(JSON.stringify(result.report)).not.toContain("present-but-never-retained"); + expect(writeReport).toHaveBeenCalledOnce(); + }); + + it("fails before GitHub access when repository or credentials are outside the bounded contract", async () => { + const ghApi = vi.fn(); + await expect(runActionsRunnerAssignmentAudit({ + env: { GH_TOKEN: "token", NOEMA_ACTIONS_AUDIT_REPOSITORY: "ContextualWisdomLab/other", NOEMA_ACTIONS_AUDIT_HEAD_SHA: expectedHead, NOEMA_ACTIONS_AUDIT_RUN_IDS: "100" }, + observed_at: "2026-08-10T00:00:00.000Z", + gh_api: ghApi, + write_report: vi.fn(), + })).rejects.toThrow("ContextualWisdomLab/noema"); + await expect(runActionsRunnerAssignmentAudit({ + env: { NOEMA_ACTIONS_AUDIT_REPOSITORY: "ContextualWisdomLab/noema", NOEMA_ACTIONS_AUDIT_HEAD_SHA: expectedHead, NOEMA_ACTIONS_AUDIT_RUN_IDS: "100" }, + observed_at: "2026-08-10T00:00:00.000Z", + gh_api: ghApi, + write_report: vi.fn(), + })).rejects.toThrow("GH_TOKEN"); + expect(ghApi).not.toHaveBeenCalled(); + }); +}); diff --git a/test/actions-runner-assignment-docs.test.ts b/test/actions-runner-assignment-docs.test.ts new file mode 100644 index 000000000..83076fbf1 --- /dev/null +++ b/test/actions-runner-assignment-docs.test.ts @@ -0,0 +1,47 @@ +import { readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; + +function readText(path: string): string { + return readFileSync(path, "utf8"); +} + +describe("runner-assignment operator documentation contract", () => { + it("keeps the operator command, doctoring, and changelog aligned", () => { + const packageJson = JSON.parse(readText("package.json")); + const doctoring = readText("docs/doctoring/actions-runner-assignment-audit.md"); + const changelog = readText("CHANGELOG.md"); + + expect(packageJson.scripts["operations:runner-assignment"]).toBe( + "node scripts/actions-runner-assignment-audit.mjs", + ); + + for (const phrase of [ + "GitHub Actions Runner-Assignment Audit Doctoring", + "ContextualWisdomLab/noema", + "NOEMA_ACTIONS_AUDIT_HEAD_SHA", + "NOEMA_ACTIONS_AUDIT_RUN_IDS", + "filter=all", + "PENDING", + "runner_assignment_stalled", + "deployment protection rules", + "jobs..needs", + "GH_HOST=github.com", + "NVIDIA_NIM_API_KEY", + "ambient proxy", + "merge", + "release", + "deployment", + "REST API endpoints for workflow runs", + "REST API endpoints for workflow jobs", + "Deployments and environments", + "Workflow syntax for GitHub Actions", + ]) { + expect(doctoring).toContain(phrase); + } + + expect(changelog).toContain("`operations:runner-assignment` audit"); + expect(changelog).toContain("runner assignment"); + expect(changelog).toContain("required Check"); + expect(changelog).toContain("formal review"); + }); +}); diff --git a/test/actions-runner-assignment-source.test.ts b/test/actions-runner-assignment-source.test.ts new file mode 100644 index 000000000..e35adf11d --- /dev/null +++ b/test/actions-runner-assignment-source.test.ts @@ -0,0 +1,70 @@ +import { describe, expect, it, vi } from "vitest"; +import { + collectRunnerAssignmentEvidence, + flattenJobPages, + MAX_SELECTED_RUNS, + parseSelectedRunIds, +} from "../scripts/lib/actions-runner-assignment-source.mjs"; + +const expectedHead = "0123456789abcdef0123456789abcdef01234567"; + +describe("GitHub Actions runner-assignment evidence source", () => { + it("parses a unique bounded run-id selection", () => { + expect(parseSelectedRunIds("101, 202,303")).toEqual([101, 202, 303]); + }); + + it.each([ + ["", "at least one"], + ["101,101", "unique"], + ["101,zero", "positive integer"], + [Array.from({ length: MAX_SELECTED_RUNS + 1 }, (_, index) => index + 1).join(","), "at most"], + ])("rejects malformed or unbounded run selection %s", (value, message) => { + expect(() => parseSelectedRunIds(value)).toThrow(message); + }); + + it("flattens all paginated jobs and rejects malformed page objects", () => { + expect(flattenJobPages([{ jobs: [{ id: 11 }] }, { jobs: [{ id: 12 }, { id: 13 }] }])).toEqual([{ id: 11 }, { id: 12 }, { id: 13 }]); + expect(() => flattenJobPages([{ jobs: [] }, { unexpected: [] }])).toThrow("jobs array"); + }); + + it("binds selected pull-request runs and every paginated job to the expected source head", async () => { + const fetchRun = vi.fn(async (runId: number) => ({ + id: runId, + name: runId === 101 ? "ci" : "reviewer-ci", + event: "pull_request", + head_sha: expectedHead, + status: "completed", + conclusion: runId === 101 ? "failure" : "success", + created_at: "2026-08-09T23:50:00.000Z", + })); + const fetchJobPages = vi.fn(async (runId: number) => [{ jobs: [{ + id: runId * 10, + name: "verify", + status: "completed", + conclusion: runId === 101 ? "failure" : "success", + started_at: "2026-08-09T23:51:00.000Z", + completed_at: "2026-08-09T23:52:00.000Z", + runner_id: 44, + runner_name: "GitHub Actions 44", + }] }]); + + await expect(collectRunnerAssignmentEvidence({ + expected_head_sha: expectedHead, + observed_at: "2026-08-10T00:00:00.000Z", + queue_grace_milliseconds: 300_000, + run_ids: [101, 202], + fetch_run: fetchRun, + fetch_job_pages: fetchJobPages, + })).resolves.toEqual({ + expected_head_sha: expectedHead, + observed_at: "2026-08-10T00:00:00.000Z", + queue_grace_milliseconds: 300_000, + runs: [ + expect.objectContaining({ id: 101, head_sha: expectedHead, jobs: [expect.objectContaining({ id: 1010, runner_id: 44 })] }), + expect.objectContaining({ id: 202, head_sha: expectedHead, jobs: [expect.objectContaining({ id: 2020, runner_id: 44 })] }), + ], + }); + expect(fetchRun).toHaveBeenCalledTimes(2); + expect(fetchJobPages).toHaveBeenCalledTimes(2); + }); +});