Repository navigation
ci: retry transient GHCR PR digest pulls #9583
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
d47a117
ci: retry transient GHCR PR digest pulls
prekshivyas cb59809
test: cover near-match GHCR pull errors
prekshivyas 5cf6b1e
Merge branch 'main' into agent/retry-ghcr-pr-digest-pulls
sandl99 418eb9d
Merge branch 'main' into agent/retry-ghcr-pr-digest-pulls
sandl99 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| #!/usr/bin/env bash | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| set -euo pipefail | ||
|
|
||
| if [ "$#" -ne 2 ]; then | ||
| echo "usage: $0 <ghcr-reference-at-digest> <platform>" >&2 | ||
| exit 2 | ||
| fi | ||
|
|
||
| reference="$1" | ||
| platform="$2" | ||
| max_attempts=5 | ||
| retry_delays=(2 4 8 16) | ||
|
|
||
| if [[ ! "$reference" =~ ^ghcr\.io/[a-z0-9._/-]+@sha256:[a-f0-9]{64}$ ]]; then | ||
| echo "ERROR: public image reference must be an exact lowercase GHCR digest" >&2 | ||
| exit 2 | ||
| fi | ||
| if [ "$platform" != "linux/amd64" ]; then | ||
| echo "ERROR: public image pull platform must be linux/amd64" >&2 | ||
| exit 2 | ||
| fi | ||
| if [ -z "${RUNNER_TEMP:-}" ] || [ ! -d "$RUNNER_TEMP" ]; then | ||
| echo "ERROR: RUNNER_TEMP must name an existing directory" >&2 | ||
| exit 2 | ||
| fi | ||
|
|
||
| anonymous_config="$(mktemp -d "$RUNNER_TEMP/managed-pr-anonymous.XXXXXX")" | ||
| attempt_log="$anonymous_config/pull.log" | ||
| chmod 700 "$anonymous_config" | ||
| trap 'rm -rf -- "$anonymous_config"' EXIT | ||
|
|
||
| for ((attempt = 1; attempt <= max_attempts; attempt += 1)); do | ||
| : >"$attempt_log" | ||
| if env -u DOCKER_AUTH_CONFIG DOCKER_CONFIG="$anonymous_config" \ | ||
| docker pull --platform "$platform" "$reference" >"$attempt_log" 2>&1; then | ||
| if [ "$attempt" -eq 1 ]; then | ||
| outcome="passed-first-attempt" | ||
| else | ||
| outcome="passed-after-retry" | ||
| fi | ||
| echo "::notice::GHCR anonymous exact-digest pull outcome=$outcome attempt=$attempt/$max_attempts" | ||
| exit 0 | ||
| else | ||
| status="$?" | ||
| fi | ||
|
|
||
| last_line="$(awk 'NF { line=$0 } END { sub(/\r$/, "", line); print line }' "$attempt_log")" | ||
| if [ "$last_line" != "ERROR: $reference: not found" ]; then | ||
| echo "::error::GHCR anonymous exact-digest pull outcome=failed-no-retry attempt=$attempt/$max_attempts docker-exit=$status" >&2 | ||
| exit "$status" | ||
| fi | ||
|
|
||
| if [ "$attempt" -eq "$max_attempts" ]; then | ||
| echo "::error::GHCR anonymous exact-digest pull outcome=exhausted attempt=$attempt/$max_attempts failure=not-found" >&2 | ||
| exit "$status" | ||
| fi | ||
|
|
||
| delay="${retry_delays[$((attempt - 1))]}" | ||
| echo "::warning::GHCR anonymous exact-digest pull outcome=transient-external attempt=$attempt/$max_attempts retry-in=${delay}s" >&2 | ||
| sleep "$delay" | ||
| done |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,156 @@ | ||
| // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| // SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| import { spawnSync } from "node:child_process"; | ||
| import fs from "node:fs"; | ||
| import os from "node:os"; | ||
| import path from "node:path"; | ||
|
|
||
| import { describe, expect, it } from "vitest"; | ||
|
|
||
| const repoRoot = path.resolve(import.meta.dirname, ".."); | ||
| const puller = path.join(repoRoot, "scripts/checks/pull-public-exact-digest.sh"); | ||
| const reference = `ghcr.io/nvidia/nemoclaw/langchain-deepagents-code-sandbox@sha256:${"a".repeat(64)}`; | ||
|
|
||
| type Scenario = "exhausted" | "near-match" | "success" | "terminal" | "transient-then-success"; | ||
|
|
||
| function runPuller(scenario: Scenario, candidateReference = reference) { | ||
| const temporaryRoot = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-public-pull-")); | ||
| const fakeBin = path.join(temporaryRoot, "bin"); | ||
| const countFile = path.join(temporaryRoot, "count"); | ||
| const configLog = path.join(temporaryRoot, "docker-configs"); | ||
| const sleepLog = path.join(temporaryRoot, "sleeps"); | ||
| fs.mkdirSync(fakeBin); | ||
| fs.writeFileSync( | ||
| path.join(fakeBin, "docker"), | ||
| `#!/usr/bin/env bash | ||
| set -euo pipefail | ||
| count=0 | ||
| if [ -f "$COUNT_FILE" ]; then | ||
| count="$(cat "$COUNT_FILE")" | ||
| fi | ||
| count=$((count + 1)) | ||
| printf '%s\n' "$count" >"$COUNT_FILE" | ||
| printf '%s\n' "$DOCKER_CONFIG" >>"$CONFIG_LOG" | ||
| [ -z "\${DOCKER_AUTH_CONFIG+x}" ] || exit 91 | ||
| [ "$*" = "pull --platform linux/amd64 $EXPECTED_REFERENCE" ] || exit 90 | ||
| if [ "$SCENARIO" = "terminal" ]; then | ||
| echo "denied: permission_denied" >&2 | ||
| exit 41 | ||
| fi | ||
| if [ "$SCENARIO" = "near-match" ]; then | ||
| echo "ERROR: $EXPECTED_REFERENCE: not found while resolving manifest" >&2 | ||
| exit 43 | ||
| fi | ||
| if [ "$SCENARIO" = "exhausted" ] || { [ "$SCENARIO" = "transient-then-success" ] && [ "$count" -eq 1 ]; }; then | ||
| echo "ERROR: $EXPECTED_REFERENCE: not found" >&2 | ||
| exit 42 | ||
| fi | ||
| echo "pulled $EXPECTED_REFERENCE" | ||
| `, | ||
| { mode: 0o755 }, | ||
| ); | ||
| fs.writeFileSync( | ||
| path.join(fakeBin, "sleep"), | ||
| '#!/usr/bin/env bash\nset -euo pipefail\nprintf \'%s\\n\' "$1" >>"$SLEEP_LOG"\n', | ||
| { mode: 0o755 }, | ||
| ); | ||
|
|
||
| try { | ||
| const result = spawnSync(puller, [candidateReference, "linux/amd64"], { | ||
| encoding: "utf8", | ||
| env: { | ||
| ...process.env, | ||
| CONFIG_LOG: configLog, | ||
| COUNT_FILE: countFile, | ||
| DOCKER_AUTH_CONFIG: "must-not-reach-docker", | ||
| EXPECTED_REFERENCE: candidateReference, | ||
| PATH: `${fakeBin}:${process.env.PATH ?? ""}`, | ||
| RUNNER_TEMP: temporaryRoot, | ||
| SCENARIO: scenario, | ||
| SLEEP_LOG: sleepLog, | ||
| }, | ||
| }); | ||
| const count = fs.existsSync(countFile) ? Number(fs.readFileSync(countFile, "utf8").trim()) : 0; | ||
| const configs = fs.existsSync(configLog) | ||
| ? fs.readFileSync(configLog, "utf8").trim().split("\n") | ||
| : []; | ||
| const sleeps = fs.existsSync(sleepLog) | ||
| ? fs.readFileSync(sleepLog, "utf8").trim().split("\n") | ||
| : []; | ||
| return { | ||
| ...result, | ||
| configs, | ||
| configsWereRemoved: configs.every((config) => !fs.existsSync(config)), | ||
| count, | ||
| sleeps, | ||
| }; | ||
| } finally { | ||
| fs.rmSync(temporaryRoot, { force: true, recursive: true }); | ||
| } | ||
| } | ||
|
|
||
| describe("pull-public-exact-digest", () => { | ||
| it("passes once with a credential-free Docker configuration", () => { | ||
| const result = runPuller("success"); | ||
|
|
||
| expect(result.status, result.stderr).toBe(0); | ||
| expect(result.count).toBe(1); | ||
| expect(result.sleeps).toEqual([]); | ||
| expect(result.configsWereRemoved).toBe(true); | ||
| expect(result.stdout).toContain("outcome=passed-first-attempt attempt=1/5"); | ||
| }); | ||
|
|
||
| it("retries the exact transient GHCR not-found result and removes anonymous state", () => { | ||
| const result = runPuller("transient-then-success"); | ||
|
|
||
| expect(result.status, result.stderr).toBe(0); | ||
| expect(result.count).toBe(2); | ||
| expect(result.sleeps).toEqual(["2"]); | ||
| expect(new Set(result.configs).size).toBe(1); | ||
| expect(result.configsWereRemoved).toBe(true); | ||
| expect(result.stderr).toContain("outcome=transient-external attempt=1/5 retry-in=2s"); | ||
| expect(result.stdout).toContain("outcome=passed-after-retry attempt=2/5"); | ||
| expect(result.stdout + result.stderr).not.toContain(`${reference}: not found`); | ||
| }); | ||
|
|
||
| it("does not retry a non-exact Docker error", () => { | ||
| const result = runPuller("terminal"); | ||
|
|
||
| expect(result.status).toBe(41); | ||
| expect(result.count).toBe(1); | ||
| expect(result.sleeps).toEqual([]); | ||
| expect(result.configsWereRemoved).toBe(true); | ||
| expect(result.stderr).toContain("outcome=failed-no-retry attempt=1/5 docker-exit=41"); | ||
| expect(result.stderr).not.toContain("permission_denied"); | ||
| }); | ||
|
|
||
| it("does not retry a near-match Docker not-found error", () => { | ||
| const result = runPuller("near-match"); | ||
|
|
||
| expect(result.status).toBe(43); | ||
| expect(result.count).toBe(1); | ||
| expect(result.sleeps).toEqual([]); | ||
| expect(result.configsWereRemoved).toBe(true); | ||
| expect(result.stderr).toContain("outcome=failed-no-retry attempt=1/5 docker-exit=43"); | ||
| expect(result.stderr).not.toContain("while resolving manifest"); | ||
| }); | ||
|
|
||
| it("fails after the bounded retry schedule is exhausted", () => { | ||
| const result = runPuller("exhausted"); | ||
|
|
||
| expect(result.status).toBe(42); | ||
| expect(result.count).toBe(5); | ||
| expect(result.sleeps).toEqual(["2", "4", "8", "16"]); | ||
| expect(result.configsWereRemoved).toBe(true); | ||
| expect(result.stderr).toContain("outcome=exhausted attempt=5/5 failure=not-found"); | ||
| }); | ||
|
|
||
| it("rejects a mutable or non-GHCR reference before Docker runs", () => { | ||
| const result = runPuller("terminal", "docker.io/nvidia/nemoclaw:latest"); | ||
|
|
||
| expect(result.status).toBe(2); | ||
| expect(result.count).toBe(0); | ||
| expect(result.stderr).toContain("must be an exact lowercase GHCR digest"); | ||
| }); | ||
| }); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.