Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
e7ecfea
test(security): require runner audit token redaction
seonghobae Aug 16, 2026
bdb4864
fix(security): redact runner audit credential errors
seonghobae Aug 16, 2026
a39f9fd
fix(security): keep token redaction coverage exact
seonghobae Aug 16, 2026
f0d4aed
test(security): cover missing gh stderr redaction path
seonghobae Aug 16, 2026
fb04907
test(operations): reproduce queued job timestamp false assignment
seonghobae Aug 16, 2026
9f2557d
fix(operations): require runner identity for assignment evidence
seonghobae Aug 16, 2026
2cd4103
test(docs): reject timestamp runner-assignment authority
seonghobae Aug 16, 2026
b4e0f99
docs(operations): align runner assignment authority
seonghobae Aug 16, 2026
5752552
test(security): require runner audit token capability
seonghobae Aug 16, 2026
fa0cefd
fix(security): broker runner audit token capability
seonghobae Aug 16, 2026
627b0cb
test(docs): require runner token capability contract
seonghobae Aug 16, 2026
c58fed9
docs(security): document runner token capability ingress
seonghobae Aug 16, 2026
61c6892
test(security): align default runner audit credential ingress
seonghobae Aug 16, 2026
bbbfe8e
fix(operations): normalize runner audit failure boundaries
seonghobae Aug 16, 2026
0634ba5
test(operations): align production runner audit capability boundary
seonghobae Aug 16, 2026
fe09e34
merge: integrate protected main after #396
seonghobae Aug 16, 2026
5f861fb
fix(operations): preserve malformed JSON evidence contract
seonghobae Aug 16, 2026
b0a7a38
merge: restack runner audit on protected main a634066
seonghobae Aug 16, 2026
c866654
fix(coverage): remove unreachable GitHub API default branch
seonghobae Aug 16, 2026
f387edf
test(security): prove delegated runner audit token wins
seonghobae Aug 16, 2026
c4a0ffb
test(security): reject ambient runner audit credential selection
seonghobae Aug 16, 2026
f4a4bc9
test: cover non-error runner audit rejection
seonghobae Aug 16, 2026
64c6213
docs(operations): record runner audit credential and assignment contract
cursoragent Aug 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

## Unreleased
- 읽기 전용 `operations:runner-assignment` audit를 추가해 exact workflow run/source head에 대한 runner assignment를 완전 pagination으로 진단하고, 신선한 unassigned queue는 bounded grace 이후 실패-폐쇄한다. 이 증빙은 runner assignment와 required Check/CI, formal review, merge, release, deployment authority를 분리하며 assigned runner 이후 workflow failure를 성공으로 승격하지 않는다.
- production `operations:runner-assignment` audit는 `NOEMA_MAINTAINER_TOKEN_PATH`의 owner-only capability file만 읽고, ambient `GH_TOKEN`만 있으면 실패-폐쇄한다. `gh` spawn/stderr 진단은 활성 토큰을 exact-match로 `[REDACTED]` 치환한다. assignment authority는 양의 `runner_id` 또는 비어 있지 않은 `runner_name`만 인정하며 queued `started_at`은 assignment evidence가 아니다. 운영자는 명령 실행 전에 capability file을 준비하고, PASS를 required Check·formal review·merge 권한으로 해석하지 마십시오.
- coordinated vulnerability disclosure 정책과 evidence-preserving vulnerability handling lifecycle, read-only private-vulnerability-reporting setting audit를 추가한다. 이 source 변경은 live private reporting 활성화·notification staffing·end-to-end advisory exercise·release/deployment authority를 증명하지 않는다.
- 개발 의존성 체인의 transitive `nanoid` lockfile resolution을 `3.3.17`에서 `3.3.18`로 최소 갱신하여 GHSA-2v37-7h3g-55p8 / CVE-2026-67213 보안 게이트를 복구한다. PostCSS의 선언 범위 `^3.3.16`과 다른 package metadata는 변경하지 않으며 audit waiver·ignore·severity 완화 없이 `npm ci`/`npm audit --audit-level=high`가 exact head에서 재검증되도록 유지한다.
- lockfile 재생성 도구 체인을 Node.js 24.19.0/npm 11.17.0으로 정확히 고정하고, `strict-allow-scripts=true` 아래 승인된 install-script identity만 실행하며 schema v3 exact-base lockfile change control로 package metadata drift를 실패-폐쇄한다. exact package before/after digest에 더해 top-level metadata digest와 대규모 package-set bulk evidence를 결합하며, 선행 `nanoid@3.3.18` 보안 수정과 explicit `npm ci --legacy-peer-deps=false --install-links=false` 계약을 보존한다. package-manager/toolchain·install-script authority·vulnerability audit·review/merge authority는 별도 증거 계층으로 유지한다.
Expand Down
27 changes: 20 additions & 7 deletions docs/doctoring/actions-runner-assignment-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,20 @@ The operator supplies an explicit bounded set of GitHub Actions workflow-run IDs
Required invariants are:

1. the repository is exactly `ContextualWisdomLab/noema`;
2. `GH_TOKEN` exists only as transport authority for read-only API calls and is never retained in the report;
2. the production CLI reads its short-lived GitHub transport authority from an owner-only delegated token capability file named by `NOEMA_MAINTAINER_TOKEN_PATH`; the bearer value is never read from ambient `GH_TOKEN` and is never retained in the report;
3. the expected source head is one canonical lowercase 40-character commit SHA;
4. one to twenty unique positive run IDs are selected explicitly;
5. selected runs must be `pull_request` runs bound to that exact source head;
6. job pages are fully paginated with `per_page=100` and `filter=all`, bounded to at most 2,000 retained jobs;
7. runner assignment is observed only from job evidence such as `started_at`, a positive `runner_id`, or a non-empty `runner_name`;
7. runner assignment is observed only from a positive `runner_id` or a non-empty `runner_name`; `started_at` is not runner-assignment authority because GitHub may populate it while a job is still queued without runner identity;
8. a `waiting`, `pending`, or `requested` job remains non-passing `PENDING` because those states do not by themselves isolate runner allocation;
9. a queued job in a run where another job has already received a runner remains non-passing `PENDING`, because the queued job may be waiting on an explicit `jobs.<job_id>.needs` dependency rather than runner capacity;
10. the bounded grace may produce `runner_assignment_stalled` only when the workflow run itself remains `queued`, the job remains `queued`, and no job in that selected run has assignment evidence;
11. an assigned job may produce runner-assignment `PASS` even if its later workflow/test conclusion is `failure`, because those are separate evidence classes.

The default runner-allocation grace is five minutes and may be bounded by `NOEMA_ACTIONS_AUDIT_QUEUE_GRACE_MILLISECONDS`; the evaluator rejects values above thirty minutes rather than allowing a true isolated queue condition to remain indefinitely pending.

This classifier is intentionally conservative because the GitHub workflow-job REST representation does not expose a durable repository-consumable timestamp meaning “this job became eligible for runner allocation.” A workflow run's `created_at` is therefore not a trustworthy age for every downstream job. The evaluator uses run age only after the selected evidence isolates the top-level queued runner-allocation boundary described above.
This classifier is intentionally conservative because the GitHub workflow-job REST representation does not expose a durable repository-consumable timestamp meaning “this job became eligible for runner allocation.” A workflow run's `created_at` is therefore not a trustworthy age for every downstream job, and a job's `started_at` is not runner-assignment authority. The evaluator uses run age only after the selected evidence isolates the top-level queued runner-allocation boundary described above.

### Pre-run waits are not runner stalls

Expand All @@ -36,17 +36,28 @@ Those states remain operationally non-passing, but they are not evidence that Gi

## Operator contract

The production command consumes an owner-only delegated token capability file. The path itself is non-secret; the token bytes must be materialized by an authorized caller and removed promptly after the audit.

Example:

```bash
export GH_TOKEN='<read-only token materialized outside repository files>'
umask 077
token_dir="$(mktemp -d)"
token_path="$token_dir/runner-audit-token"
printf '%s' '<short-lived read-only token>' >"$token_path"
chmod 0600 "$token_path"
trap 'rm -rf "$token_dir"' EXIT

export NOEMA_MAINTAINER_TOKEN_PATH="$token_path"
export NOEMA_ACTIONS_AUDIT_REPOSITORY='ContextualWisdomLab/noema'
export NOEMA_ACTIONS_AUDIT_HEAD_SHA='<exact current PR source head>'
export NOEMA_ACTIONS_AUDIT_RUN_IDS='31343034891,31343034896,31343034900'

npm run operations:runner-assignment
```

An ambient `GH_TOKEN` is intentionally not a production credential source for this command. The descriptor-safe shared capability reader rejects symlinks, non-regular files, group/world-readable files, wrong-owner files where UID inspection is available, oversized content, invalid UTF-8, control-bearing tokens, and file identity/version changes during the bounded read.

The command writes the fixed report path:

```text
Expand All @@ -55,7 +66,7 @@ artifacts/operations/actions-runner-assignment-audit.json

The report records repository, expected head, selected run IDs, observation time, queue grace, deterministic checks/failures, and explicit false authority flags for required-check success, review, merge, release, and deployment. Temporary report bytes are created owner-only and atomically renamed onto the fixed report path.

`PASS` exits zero. `PENDING` and `FAIL` both exit nonzero. A malformed source identity, cross-repository request, missing credential, malformed API JSON, GitHub CLI failure, pagination-shape failure, excessive evidence, or head mismatch fails closed.
`PASS` exits zero. `PENDING` and `FAIL` both exit nonzero. A malformed source identity, cross-repository request, missing/unsafe capability file, malformed API JSON, GitHub CLI failure, pagination-shape failure, excessive evidence, or head mismatch fails closed.

## RCA interpretation

Expand All @@ -69,9 +80,9 @@ This separation matters for issue #30 because historical Noema runs exhibited qu

## Security and privacy

The collector uses GitHub Actions REST **read** endpoints only. It does not rerun, cancel, dispatch, approve, merge, modify refs, or change settings. The report does not contain `GH_TOKEN`, repository secrets, workflow logs, source contents, personal data beyond ordinary GitHub workflow/job metadata needed for the operational decision, or model output.
The collector uses GitHub Actions REST **read** endpoints only. It does not rerun, cancel, dispatch, approve, merge, modify refs, or change settings. The report does not contain the delegated token, repository secrets, workflow logs, source contents, personal data beyond ordinary GitHub workflow/job metadata needed for the operational decision, or model output.

The `gh` child process also receives a purpose-built minimal environment rather than ambient process state: only `PATH`, the read-only `GH_TOKEN`, pinned `GH_HOST=github.com`, and `NO_COLOR=1` cross the process boundary. `GITHUB_TOKEN`, `NVIDIA_NIM_API_KEY`, Maintainer/Reviewer App private material, `HOME`, and ambient proxy variables are excluded. This prevents a read-only diagnostic subprocess from accidentally inheriting stronger publication/model credentials or redirecting credential-bearing requests through an unreviewed proxy path.
The production CLI reads bearer authority only through `NOEMA_MAINTAINER_TOKEN_PATH`. The `gh` child process then receives a purpose-built minimal environment: only `PATH`, the in-memory read-only `GH_TOKEN` required by the GitHub CLI, pinned `GH_HOST=github.com`, and `NO_COLOR=1` cross that child-process boundary. `GITHUB_TOKEN`, ambient `GH_TOKEN`, `NVIDIA_NIM_API_KEY`, Maintainer/Reviewer App private material, `HOME`, and ambient proxy variables are excluded. This prevents the read-only diagnostic subprocess from inheriting stronger publication/model credentials or redirecting credential-bearing requests through an unreviewed proxy path.

The audit is diagnostic evidence. A passing assignment audit cannot satisfy branch protection, required checks, formal review, security scanning, release provenance, production deployment, or acquisition evidence.

Expand All @@ -80,6 +91,8 @@ The audit is diagnostic evidence. A passing assignment audit cannot satisfy bran
The repository-owned slice is acceptable when:

- realistic tests reproduce an isolated runner stall, a fresh queue, deployment/environment waiting, downstream dependency waiting, assigned-but-failed jobs, head mismatch, malformed evidence, pagination, and bounded selection;
- queued `started_at` timestamps without runner identity remain unassigned, while a positive `runner_id` or a non-empty `runner_name` is assignment evidence;
- the production entrypoint succeeds with an owner-only delegated token capability file and fails closed when only ambient `GH_TOKEN` is present;
- environment-protected and dependency-blocked jobs remain nonzero `PENDING` and are not mislabeled as runner-allocation stalls;
- the pure evaluator and bounded source collector are GREEN;
- the operator adapter performs only the two documented read families and fully paginates jobs;
Expand Down
47 changes: 42 additions & 5 deletions scripts/actions-runner-assignment-audit.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import {
collectRunnerAssignmentEvidence,
parseSelectedRunIds,
} from "./lib/actions-runner-assignment-source.mjs";
import { readDelegatedGithubToken } from "./lib/delegated-github-token.mjs";
import { hasDuplicateJsonObjectKeys } from "./normalize-commercial-readiness-evidence.mjs";

const AUDITED_REPOSITORY = "ContextualWisdomLab/noema";
Expand Down Expand Up @@ -51,6 +52,11 @@ function boundedErrorText(value) {
return text.replace(/[\u0000-\u001f\u007f]/g, " ").trim().slice(0, 1000);
}

function redactExactSecret(value, secret) {
const text = typeof value === "string" ? value : String(value ?? "");
return text.split(secret).join("[REDACTED]");

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

text.split(secret).join("[REDACTED]") is correct for a non-empty active token, and createGhSubprocessEnvironment currently rejects empty GH_TOKEN before this runs.

If this helper is reused with an empty secret, split("") inserts [REDACTED] between every character. Guard with if (typeof secret !== "string" || secret.length === 0) return text so a later caller cannot take that landmine.

}

/**
* Build the least-privilege environment inherited by the `gh` subprocess.
*
Expand Down Expand Up @@ -159,19 +165,25 @@ export function ghApi(path, options = {}, runtime = defaultGhRuntime) {
}
args.push(path);

const subprocessEnvironment = createGhSubprocessEnvironment(runtime.environment ?? process.env);
const result = runtime.spawn_sync("gh", args, {
timeout: GH_API_TIMEOUT_MILLISECONDS,
maxBuffer: GH_API_MAX_BUFFER_BYTES,
env: createGhSubprocessEnvironment(runtime.environment ?? process.env),
env: subprocessEnvironment,
stdio: ["ignore", "pipe", "pipe"],
});

if (result.error) {
throw new Error(`GitHub Actions evidence read failed: ${boundedErrorText(result.error.message)}`);
throw new Error(
`GitHub Actions evidence read failed: ${boundedErrorText(redactExactSecret(result.error.message, subprocessEnvironment.GH_TOKEN))}`,
);
}
if (result.status !== 0) {
const diagnostic = boundedErrorText(
redactExactSecret(result.stderr, subprocessEnvironment.GH_TOKEN),
);
throw new Error(
`GitHub Actions evidence read failed with gh exit ${result.status}: ${boundedErrorText(result.stderr)}`,
`GitHub Actions evidence read failed with gh exit ${result.status}:${diagnostic ? ` ${diagnostic}` : ""}`,
);
}

Expand Down Expand Up @@ -344,14 +356,39 @@ export async function runActionsRunnerAssignmentAudit(input) {
/**
* Execute the CLI with injectable boundaries while preserving production defaults.
*
* Production reads the delegated bearer credential only from the repository's
* descriptor-safe capability-file boundary. Tests may inject a pre-authenticated
* reader and explicit environment without requiring filesystem credential access.
*
* @param {object} options Runtime overrides used only by tests/operators.
* @returns {Promise<{exit_code: number, report: object}>} Audit result.
*/
export async function main(options = {}) {
const sourceEnvironment = options.env ?? process.env;
let auditEnvironment = sourceEnvironment;
let githubApi = options.gh_api;

if (githubApi === undefined) {
const tokenPath = String(sourceEnvironment.NOEMA_MAINTAINER_TOKEN_PATH ?? "").trim();
const delegatedToken = readDelegatedGithubToken(tokenPath);
const subprocessEnvironment = {
PATH: sourceEnvironment.PATH,
GH_TOKEN: delegatedToken,
};
auditEnvironment = {
...sourceEnvironment,
GH_TOKEN: delegatedToken,
};
githubApi = (path, apiOptions) => ghApi(path, apiOptions, {
spawn_sync: spawnSync,
environment: subprocessEnvironment,
});
}

const result = await runActionsRunnerAssignmentAudit({
env: options.env ?? process.env,
env: auditEnvironment,
observed_at: options.observed_at ?? new Date().toISOString(),
gh_api: options.gh_api ?? ghApi,
gh_api: githubApi,
write_report: options.write_report ?? writeReportAtomically,
});
const writeOutput = options.write_output ?? ((value) => process.stdout.write(value));
Expand Down
15 changes: 8 additions & 7 deletions scripts/lib/actions-runner-assignment-audit.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,9 @@ function boundedName(value) {
}

function assignmentObserved(job) {
const startedAt = parseTimestamp(job?.started_at);
const runnerId = job?.runner_id;
const runnerName = typeof job?.runner_name === "string" ? job.runner_name.trim() : "";
return startedAt !== null || positiveSafeInteger(runnerId) || runnerName.length > 0;
return positiveSafeInteger(runnerId) || runnerName.length > 0;
}

function invalidEvidence(detail) {
Expand All @@ -51,9 +50,11 @@ function invalidEvidence(detail) {
*
* The evaluator answers only whether each selected pull-request workflow job
* obtained a runner. A later test, security, or workflow conclusion remains a
* separate evidence class. Freshly queued jobs remain non-passing `PENDING`.
* A grace-window stall is emitted only when both the workflow run and the job
* remain queued with no assignment observed anywhere in the selected run;
* separate evidence class. GitHub may populate `started_at` while a queued job
* still has runner_id=0 and no runner_name, so timestamps are not assignment
* authority. Freshly queued jobs remain non-passing `PENDING`. A grace-window
* stall is emitted only when both the workflow run and the job remain queued
* with no runner identity observed anywhere in the selected run;
* protection/dependency waits stay non-passing without being mislabeled as a
* runner-allocation failure.
*
Expand Down Expand Up @@ -190,7 +191,7 @@ export function evaluateRunnerAssignmentEvidence(evidence) {
check(
"runner_assignment_observed",
true,
"GitHub job evidence shows that a runner was assigned; the later job conclusion remains separate.",
"GitHub job evidence contains a positive runner id or non-empty runner name; the later job conclusion remains separate.",
jobContext,
),
);
Expand All @@ -202,7 +203,7 @@ export function evaluateRunnerAssignmentEvidence(evidence) {
failures.push(
failure(
"runner_assignment_not_observed",
"The job reached a non-queue state without trustworthy started_at or runner identity evidence.",
"The job reached a non-queue state without trustworthy runner identity evidence.",
{ ...jobContext, job_status: jobStatus },
),
);
Expand Down
21 changes: 21 additions & 0 deletions test/actions-runner-assignment-audit.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,27 @@ describe("GitHub Actions runner-assignment evidence", () => {
expect(failureCodes(result)).toContain("runner_assignment_stalled");
});

it("does not mistake GitHub's queued started_at timestamp for runner assignment", () => {
const result = evaluate([workflowRun({
jobs: [{
id: 201,
name: "verify",
status: "queued",
conclusion: null,
started_at: "2026-08-09T23:50:00.000Z",
completed_at: null,
runner_id: 0,
runner_name: "",
}],
})]);

expect(result.status).toBe("FAIL");
expect(failureCodes(result)).toContain("runner_assignment_stalled");
expect(result.checks).not.toContainEqual(
expect.objectContaining({ code: "runner_assignment_observed", job_id: 201 }),
);
});

it("keeps a recently queued unassigned job pending rather than calling it healthy", () => {
const result = evaluate([workflowRun({ created_at: "2026-08-09T23:58:00.000Z" })]);
expect(result.status).toBe("PENDING");
Expand Down
Loading
Loading