Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 1 addition & 3 deletions .github/workflows/managed-images.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -792,9 +792,7 @@ jobs:
echo "ERROR: published PR manifest bytes do not match the build digest" >&2
exit 1
}
anonymous_config="$(mktemp -d "$RUNNER_TEMP/managed-pr-anonymous.XXXXXX")"
trap 'rm -rf -- "$anonymous_config"' EXIT
DOCKER_CONFIG="$anonymous_config" docker pull --platform linux/amd64 "$reference"
scripts/checks/pull-public-exact-digest.sh "$reference" linux/amd64
release="v$(node -p 'require("./package.json").version')"
contract_dir="$RUNNER_TEMP/managed-pr-contract"
mkdir -p "$contract_dir"
Expand Down
64 changes: 64 additions & 0 deletions scripts/checks/pull-public-exact-digest.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

set -euo pipefail

if [ "$#" -ne 2 ]; then
echo "usage: $0 <ghcr-reference-at-digest> <platform>" >&2
exit 2
fi

reference="$1"
platform="$2"
max_attempts=5
retry_delays=(2 4 8 16)

if [[ ! "$reference" =~ ^ghcr\.io/[a-z0-9._/-]+@sha256:[a-f0-9]{64}$ ]]; then
echo "ERROR: public image reference must be an exact lowercase GHCR digest" >&2
exit 2
fi
if [ "$platform" != "linux/amd64" ]; then
echo "ERROR: public image pull platform must be linux/amd64" >&2
exit 2
fi
if [ -z "${RUNNER_TEMP:-}" ] || [ ! -d "$RUNNER_TEMP" ]; then
echo "ERROR: RUNNER_TEMP must name an existing directory" >&2
exit 2
fi

anonymous_config="$(mktemp -d "$RUNNER_TEMP/managed-pr-anonymous.XXXXXX")"
attempt_log="$anonymous_config/pull.log"
chmod 700 "$anonymous_config"
trap 'rm -rf -- "$anonymous_config"' EXIT

for ((attempt = 1; attempt <= max_attempts; attempt += 1)); do
: >"$attempt_log"
if env -u DOCKER_AUTH_CONFIG DOCKER_CONFIG="$anonymous_config" \
docker pull --platform "$platform" "$reference" >"$attempt_log" 2>&1; then
if [ "$attempt" -eq 1 ]; then
outcome="passed-first-attempt"
else
outcome="passed-after-retry"
fi
echo "::notice::GHCR anonymous exact-digest pull outcome=$outcome attempt=$attempt/$max_attempts"
exit 0
else
status="$?"
fi

last_line="$(awk 'NF { line=$0 } END { sub(/\r$/, "", line); print line }' "$attempt_log")"
if [ "$last_line" != "ERROR: $reference: not found" ]; then
echo "::error::GHCR anonymous exact-digest pull outcome=failed-no-retry attempt=$attempt/$max_attempts docker-exit=$status" >&2
exit "$status"
fi

if [ "$attempt" -eq "$max_attempts" ]; then
echo "::error::GHCR anonymous exact-digest pull outcome=exhausted attempt=$attempt/$max_attempts failure=not-found" >&2
exit "$status"
fi

delay="${retry_delays[$((attempt - 1))]}"
echo "::warning::GHCR anonymous exact-digest pull outcome=transient-external attempt=$attempt/$max_attempts retry-in=${delay}s" >&2
sleep "$delay"
done
1 change: 1 addition & 0 deletions test/e2e/RETRY_INVENTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ Exhaustion remains failed.
| `hosted-runner-recovery` | Confirmed GitHub-hosted runner loss; `tools/e2e/hosted-runner-recovery.mts`, `tools/e2e/hosted-runner-loss*.mts` | Authenticated runner-allocation or internal-runner evidence that remains identical across 2 consecutive reads | 2 immediate evidence reads and at most 1 recovery request; no delay | GitHub reruns a workflow attempt | GitHub Actions | Dedicated runner-loss classifications | Source and recovery run links plus authenticated job evidence | External owner; governed by #7146, not this policy |
| `pr-rerun-reconciliation` | PR E2E dispatch reconciliation; `tools/e2e/pr-e2e-dispatch-reconciliation.mts`, `tools/e2e/pr-e2e-retry-receipt.mts` | Trusted dispatch receipt state | Contract-defined single reconciliation | Reconciles workflow and commit identity before action | GitHub Actions | Receipt-specific terminal states | Signed workflow identity and receipt | External scope; governed by #7206 |
| `github-publication-read` | GitHub API reads; `tools/e2e/base-image-publication.mts` | Fetch error, 408, rate limit, or 5xx | 3 attempts; Retry-After/rate-limit reset or linear delay capped at 10s | Read-only | GitHub API | Returned parsed selection on success; thrown terminal HTTP/fetch error on failure or exhaustion | Caller artifact records the returned publication selection; terminal errors identify exhausted fetch or HTTP status without response content | Eligible bounded read; existing implementation retained |
| `ghcr-pr-exact-digest-visibility` | Anonymous pull after same-repository PR publication; `scripts/checks/pull-public-exact-digest.sh`, `.github/workflows/managed-images.yaml` | Final nonempty Docker output line is exactly `ERROR: <validated-reference>: not found`; every other Docker failure is terminal | 5 attempts; exponential 2s, 4s, 8s, then 16s | Read-only registry request for one immutable digest; any local image-cache write repeats the same content-addressed result | GHCR | `passed-first-attempt`, `passed-after-retry`, `failed-no-retry`, or `exhausted` | Sanitized outcome, attempt number, total attempts, retry delay or Docker exit status; raw Docker output, credentials, and environment values are excluded | Eligible bounded external read after the independently fetched manifest bytes match the publication digest; every non-exact failure remains terminal and no workflow rerun occurs |
| `trusted-controller-collaborator-permission-read` | Collaborator-permission reads for manual PR dispatch and Launchable E2E dispatch; `.github/workflows/e2e.yaml` | Curl exit 5, 6, 7, 16, 18, 28, 35, 52, 55, 56, 92, 95, or 96; HTTP 408, 429, or 5xx | 3 attempts; linear 1s then 2s | Read-only GitHub API request | GitHub API | Transient API read versus terminal authentication, authorization, actor, or response failure | Operation name, attempt number, and sanitized failure class or HTTP status; no response body, header, or token | Eligible bounded read; HTTP 401, 403, 404, and 422, malformed responses, actor failures, and insufficient roles remain terminal; no cached permission or workflow rerun |
| `pr-exact-openclaw-mcp-repetition` | Complete OpenClaw trusted-private MCP bridge acceptance; `.github/workflows/managed-images.yaml`, `test/e2e/live/mcp-bridge.test.ts` | Either independent matrix execution fails | 2 required executions on fresh runners; 0 workflow or test retries | Each execution creates and cleans up its own sandbox against the same exact candidate publication cohort | NemoClaw | Each execution passes or fails independently; both must pass | Existing redacted MCP diagnostics, request ledger, cleanup evidence, and fixture-credential scan for each matrix pass | Fixed acceptance repetition required by #8746; not a retry, and one pass never masks the other |
| `github-exact-artifact-content-read` | Bound base-image or PR managed-image contract artifact; `tools/e2e/exact-artifact-download.mts`, `tools/e2e/pr-managed-image-publication.mts` | Transport failure, HTTP 408, HTTP 429, or HTTP 5xx while reading one pre-bound artifact ID | 3 attempts; Retry-After or linear delay capped at 10s | Read-only request against one immutable artifact ID, name, size, digest, producer run, attempt, and producer commit | GitHub artifact service | `passed-first-attempt`, `passed-after-retry`, `exhausted` for transient exhaustion, or `failed-no-retry` for terminal HTTP; identity, size, digest, archive, and contract failures throw without an aggregate outcome or `failureClass` | Content-read attempts log only the sanitized operation, attempt, HTTP status or transport class, and outcome; thrown validation failures expose only their bounded error message, never headers, body, token, signed URL, or artifact content | Standalone bounded content read; it does not use `retry-policy.ts` or `RetryEvidence`, and all identity, integrity, archive, and contract failures remain terminal |
Expand Down
14 changes: 14 additions & 0 deletions test/helpers/vitest-watch-triggers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,13 @@ export const vitestWatchTriggerPatterns: VitestWatchTriggerPattern[] = [
"test/dcode-base-image-workflow.test.ts",
),
},
{
pattern: /(?:^|\/)\.github\/workflows\/managed-images\.yaml$/,
testsToRun: runTests(
"test/managed-image-publication-workflow.test.ts",
"test/pull-public-exact-digest.test.ts",
),
},
{
pattern: /(?:^|\/)\.github\/actions\/build-base-image-platform\/action\.yaml$/,
testsToRun: runTests(
Expand Down Expand Up @@ -142,6 +149,13 @@ export const vitestWatchTriggerPatterns: VitestWatchTriggerPattern[] = [
"test/dcode-base-image-workflow.test.ts",
),
},
{
pattern: /(?:^|\/)scripts\/checks\/pull-public-exact-digest\.sh$/,
testsToRun: runTests(
"test/pull-public-exact-digest.test.ts",
"test/managed-image-publication-workflow.test.ts",
),
},
{
pattern: /(?:^|\/)scripts\/e2e\/sanitize-trace-timing\.py$/,
testsToRun: runTests(
Expand Down
2 changes: 1 addition & 1 deletion test/managed-image-publication-workflow.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -589,7 +589,7 @@ describe("complete managed-image publication workflow", () => {
expect(exportContract.if).toBe(sameRepository);
expect(uploadContract.if).toBe(sameRepository);
expect(steps.indexOf(logout)).toBeLessThan(steps.indexOf(exportContract));
expect(exportContract.run).toContain('DOCKER_CONFIG="$anonymous_config" docker pull');
expect(exportContract.run).toContain("scripts/checks/pull-public-exact-digest.sh");
expect(exportContract.run).toContain("revision: $revision");
expect(JSON.stringify(prBuilder).match(/secrets\.GITHUB_TOKEN/gu)).toHaveLength(1);
expect(JSON.stringify(prBuilder)).not.toContain("github.token");
Expand Down
156 changes: 156 additions & 0 deletions test/pull-public-exact-digest.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

import { spawnSync } from "node:child_process";
import fs from "node:fs";
import os from "node:os";
import path from "node:path";

import { describe, expect, it } from "vitest";

const repoRoot = path.resolve(import.meta.dirname, "..");
const puller = path.join(repoRoot, "scripts/checks/pull-public-exact-digest.sh");
const reference = `ghcr.io/nvidia/nemoclaw/langchain-deepagents-code-sandbox@sha256:${"a".repeat(64)}`;

type Scenario = "exhausted" | "near-match" | "success" | "terminal" | "transient-then-success";

function runPuller(scenario: Scenario, candidateReference = reference) {
const temporaryRoot = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-public-pull-"));
const fakeBin = path.join(temporaryRoot, "bin");
const countFile = path.join(temporaryRoot, "count");
const configLog = path.join(temporaryRoot, "docker-configs");
const sleepLog = path.join(temporaryRoot, "sleeps");
fs.mkdirSync(fakeBin);
fs.writeFileSync(
path.join(fakeBin, "docker"),
`#!/usr/bin/env bash
set -euo pipefail
count=0
if [ -f "$COUNT_FILE" ]; then
count="$(cat "$COUNT_FILE")"
fi
count=$((count + 1))
printf '%s\n' "$count" >"$COUNT_FILE"
printf '%s\n' "$DOCKER_CONFIG" >>"$CONFIG_LOG"
[ -z "\${DOCKER_AUTH_CONFIG+x}" ] || exit 91
[ "$*" = "pull --platform linux/amd64 $EXPECTED_REFERENCE" ] || exit 90
if [ "$SCENARIO" = "terminal" ]; then
echo "denied: permission_denied" >&2
exit 41
fi
if [ "$SCENARIO" = "near-match" ]; then
echo "ERROR: $EXPECTED_REFERENCE: not found while resolving manifest" >&2
exit 43
fi
if [ "$SCENARIO" = "exhausted" ] || { [ "$SCENARIO" = "transient-then-success" ] && [ "$count" -eq 1 ]; }; then
echo "ERROR: $EXPECTED_REFERENCE: not found" >&2
exit 42
fi
echo "pulled $EXPECTED_REFERENCE"
`,
{ mode: 0o755 },
);
fs.writeFileSync(
path.join(fakeBin, "sleep"),
'#!/usr/bin/env bash\nset -euo pipefail\nprintf \'%s\\n\' "$1" >>"$SLEEP_LOG"\n',
{ mode: 0o755 },
);

try {
const result = spawnSync(puller, [candidateReference, "linux/amd64"], {
encoding: "utf8",
env: {
...process.env,
CONFIG_LOG: configLog,
COUNT_FILE: countFile,
DOCKER_AUTH_CONFIG: "must-not-reach-docker",
EXPECTED_REFERENCE: candidateReference,
PATH: `${fakeBin}:${process.env.PATH ?? ""}`,
RUNNER_TEMP: temporaryRoot,
SCENARIO: scenario,
SLEEP_LOG: sleepLog,
},
});
const count = fs.existsSync(countFile) ? Number(fs.readFileSync(countFile, "utf8").trim()) : 0;
const configs = fs.existsSync(configLog)
? fs.readFileSync(configLog, "utf8").trim().split("\n")
: [];
const sleeps = fs.existsSync(sleepLog)
? fs.readFileSync(sleepLog, "utf8").trim().split("\n")
: [];
return {
...result,
configs,
configsWereRemoved: configs.every((config) => !fs.existsSync(config)),
count,
sleeps,
};
} finally {
fs.rmSync(temporaryRoot, { force: true, recursive: true });
}
}

describe("pull-public-exact-digest", () => {
it("passes once with a credential-free Docker configuration", () => {
const result = runPuller("success");

expect(result.status, result.stderr).toBe(0);
expect(result.count).toBe(1);
expect(result.sleeps).toEqual([]);
expect(result.configsWereRemoved).toBe(true);
expect(result.stdout).toContain("outcome=passed-first-attempt attempt=1/5");
});

it("retries the exact transient GHCR not-found result and removes anonymous state", () => {
const result = runPuller("transient-then-success");

expect(result.status, result.stderr).toBe(0);
expect(result.count).toBe(2);
expect(result.sleeps).toEqual(["2"]);
expect(new Set(result.configs).size).toBe(1);
expect(result.configsWereRemoved).toBe(true);
expect(result.stderr).toContain("outcome=transient-external attempt=1/5 retry-in=2s");
expect(result.stdout).toContain("outcome=passed-after-retry attempt=2/5");
expect(result.stdout + result.stderr).not.toContain(`${reference}: not found`);
});

it("does not retry a non-exact Docker error", () => {
const result = runPuller("terminal");

expect(result.status).toBe(41);
expect(result.count).toBe(1);
expect(result.sleeps).toEqual([]);
expect(result.configsWereRemoved).toBe(true);
expect(result.stderr).toContain("outcome=failed-no-retry attempt=1/5 docker-exit=41");
expect(result.stderr).not.toContain("permission_denied");
});
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

it("does not retry a near-match Docker not-found error", () => {
const result = runPuller("near-match");

expect(result.status).toBe(43);
expect(result.count).toBe(1);
expect(result.sleeps).toEqual([]);
expect(result.configsWereRemoved).toBe(true);
expect(result.stderr).toContain("outcome=failed-no-retry attempt=1/5 docker-exit=43");
expect(result.stderr).not.toContain("while resolving manifest");
});

it("fails after the bounded retry schedule is exhausted", () => {
const result = runPuller("exhausted");

expect(result.status).toBe(42);
expect(result.count).toBe(5);
expect(result.sleeps).toEqual(["2", "4", "8", "16"]);
expect(result.configsWereRemoved).toBe(true);
expect(result.stderr).toContain("outcome=exhausted attempt=5/5 failure=not-found");
});

it("rejects a mutable or non-GHCR reference before Docker runs", () => {
const result = runPuller("terminal", "docker.io/nvidia/nemoclaw:latest");

expect(result.status).toBe(2);
expect(result.count).toBe(0);
expect(result.stderr).toContain("must be an exact lowercase GHCR digest");
});
});
9 changes: 9 additions & 0 deletions test/vitest-watch-triggers.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,11 @@ describe("Vitest opaque-input watch triggers", () => {
"test/managed-image-publication-workflow.test.ts",
"test/dcode-base-image-workflow.test.ts",
]);
expect(triggeredBy(".github/workflows/managed-images.yaml")).toEqual([
"test/pi-candidate-runtime-artifacts.test.ts",
"test/managed-image-publication-workflow.test.ts",
"test/pull-public-exact-digest.test.ts",
]);
expect(triggeredBy("scripts/export-managed-base-image-contract.sh")).toEqual([
"test/managed-base-image-contract.test.ts",
"test/managed-image-publication-workflow.test.ts",
Expand All @@ -165,6 +170,10 @@ describe("Vitest opaque-input watch triggers", () => {
"test/managed-image-publication-workflow.test.ts",
"test/dcode-base-image-workflow.test.ts",
]);
expect(triggeredBy("scripts/checks/pull-public-exact-digest.sh")).toEqual([
"test/pull-public-exact-digest.test.ts",
"test/managed-image-publication-workflow.test.ts",
]);
expect(triggeredBy("scripts/e2e/sanitize-trace-timing.py")).toEqual([
"test/e2e/support/e2e-scorecard.test.ts",
"test/e2e/support/sanitize-trace-timing.test.ts",
Expand Down
Loading