ci: retry cosign sign on transient GHCR/Rekor failures #2100
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,87 @@ | ||||||
| #!/usr/bin/env bash | ||||||
| # Retry cosign sign on transient registry / Rekor / Fulcio failures. | ||||||
| # | ||||||
| # `cosign sign` against a published digest is idempotent: signing the | ||||||
| # same digest twice either succeeds again or hits a Rekor | ||||||
| # `createLogEntryConflict` (already-logged) response, which callers | ||||||
| # treat as success. Transient GHCR/Rekor/Fulcio errors (5xx, 429, TLS | ||||||
| # handshake stalls, connection resets) almost always settle inside the | ||||||
| # next attempt window, so a bounded retry turns a noisy infra blip | ||||||
| # into a green run instead of failing the whole Docker workflow. | ||||||
| # | ||||||
| # Usage: | ||||||
| # cosign_sign_with_retry.sh <ref> | ||||||
| # <ref> is the full image reference, e.g. | ||||||
| # ghcr.io/aureliolo/synthorg-sandbox-base@sha256:abc... | ||||||
| # | ||||||
| # Behaviour: | ||||||
| # - Captures combined stdout+stderr of `cosign sign --yes <ref>`. | ||||||
| # - On exit 0: prints captured output, exits 0. | ||||||
| # - On exit non-0: | ||||||
| # * Output contains `createLogEntryConflict` -> already signed, | ||||||
| # emit `::notice::` and exit 0 (preserves the idempotency branch | ||||||
| # the inline shell blocks used to carry). | ||||||
| # * Output matches the shared transient regex sourced from | ||||||
| # docker_push_with_retry.sh (single source of truth for | ||||||
| # "is this a registry-side flake?") -> warn + sleep + retry | ||||||
| # with exponential backoff. | ||||||
| # * Otherwise -> non-transient cosign / Rekor / Fulcio error, | ||||||
| # surface output and exit with cosign's exit code. | ||||||
| # - Final attempt with no terminal classification: surface all | ||||||
| # output and exit with cosign's exit code. | ||||||
| set -euo pipefail | ||||||
|
|
||||||
| REF="${1:?usage: cosign_sign_with_retry.sh <ref>}" | ||||||
|
|
||||||
| # Same regex the docker push helper uses; `--print-transient-re` keeps | ||||||
| # both scripts in lockstep so a new transient signature added in one | ||||||
| # place automatically protects every signing call too. | ||||||
| SCRIPT_DIR="$(dirname "$(readlink -f "$0")")" | ||||||
| TRANSIENT_RE="$("$SCRIPT_DIR/docker_push_with_retry.sh" --print-transient-re)" | ||||||
|
|
||||||
| # 4 attempts, backoff 15s -> 30s -> 60s = ~1m45s of wait in the worst | ||||||
| # case before the final attempt. Matches docker_push_with_retry.sh so | ||||||
| # cosign rides through GHCR's typical 30-90s unicorn windows under the | ||||||
| # same budget the push step already does. | ||||||
| ATTEMPTS=4 | ||||||
| BACKOFF=15 | ||||||
|
|
||||||
| for ((i = 1; i <= ATTEMPTS; i++)); do | ||||||
| out="" | ||||||
| rc=0 | ||||||
| out="$(cosign sign --yes "$REF" 2>&1)" || rc=$? | ||||||
| if [ "$rc" -eq 0 ]; then | ||||||
| printf '%s\n' "$out" | ||||||
| exit 0 | ||||||
| fi | ||||||
|
|
||||||
| # Idempotency branch: a re-sign that lost the createLogEntry race is | ||||||
| # success, not a transient error. Check this BEFORE the regex so | ||||||
| # attempt 1 -> 5xx -> attempt 2 -> conflict resolves cleanly. | ||||||
| if printf '%s' "$out" | grep -q 'createLogEntryConflict'; then | ||||||
|
Contributor
There was a problem hiding this comment. Using
Suggested change
|
||||||
| printf '%s\n' "$out" | ||||||
| echo "::notice::Image ${REF} already signed -- skipping" | ||||||
| exit 0 | ||||||
| fi | ||||||
|
|
||||||
| if [ "$i" -eq "$ATTEMPTS" ]; then | ||||||
| printf '%s\n' "$out" | ||||||
| echo "::error::cosign sign ${REF} failed after ${ATTEMPTS} attempts (last exit ${rc})" >&2 | ||||||
| exit "$rc" | ||||||
| fi | ||||||
|
|
||||||
| if printf '%s' "$out" | grep -qiE "$TRANSIENT_RE"; then | ||||||
|
Contributor
There was a problem hiding this comment. If
Suggested change
|
||||||
| printf '%s\n' "$out" >&2 | ||||||
| echo "::warning::cosign sign ${REF} hit transient error (attempt ${i}/${ATTEMPTS}, rc=${rc}); sleeping ${BACKOFF}s before retry" >&2 | ||||||
| sleep "$BACKOFF" | ||||||
| BACKOFF=$((BACKOFF * 2)) | ||||||
| continue | ||||||
| fi | ||||||
|
|
||||||
| # Non-transient: surface output and bubble up immediately. Auth | ||||||
| # denials, malformed digests, Rekor schema rejections, etc. will | ||||||
| # never improve on a retry. | ||||||
| printf '%s\n' "$out" | ||||||
| echo "::error::cosign sign ${REF} failed with non-transient error (exit ${rc}); not retrying" >&2 | ||||||
| exit "$rc" | ||||||
| done | ||||||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For robustness in CI environments where file permissions might not be preserved or correctly set in the repository, it is safer to explicitly invoke the helper script using
bash. This ensures the script runs even if the execute bit is missing.