From ab4ca1db46afe2bbd809e713e89b8c3ac76401fa Mon Sep 17 00:00:00 2001 From: Sandy Chapman Date: Wed, 29 Jul 2026 12:49:47 -0300 Subject: [PATCH 1/3] ci: cache the pinned OPA binary and retry its download MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Build OPA policy WASM` is a required job, and it has been failing on an unretried download from openpolicyagent.org — twice in three attempts on a single PR, each time taking `Python e2e tests` (which needs its artifact) down with it. `script/build_policy_wasm.sh` already prefers a cached binary at `.cache/opa//` before downloading, but nothing in CI ever populated that directory, so every run depended on the network. The composite action now restores and saves it, keyed on the pinned version, so the fetch is only needed the first time a version is seen. The version is parsed out of the script rather than duplicated, so the key cannot drift from the pin; parsing (not sourcing) because executing the script would build the WASM a second time, and before the cache is restored. Both `curl` calls also gain `--retry 3 --retry-delay 2 --retry-all-errors`. `--retry-all-errors` matters here: the observed failures were connection-level, which plain `--retry` does not cover. Verified locally: cold cache downloads and populates `.cache/opa//`; a second run builds with no download at all. The retry flags were checked against an unreachable endpoint (4 attempts, backoff between). Regenerating policy.wasm leaves the tree clean, so the output stays reproducible. Signed-off-by: Sandy Chapman --- .github/actions/build-policy-wasm/action.yaml | 27 +++++++++++++++++++ script/build_policy_wasm.sh | 4 +-- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/.github/actions/build-policy-wasm/action.yaml b/.github/actions/build-policy-wasm/action.yaml index 863406d262e..d7a44538bc2 100644 --- a/.github/actions/build-policy-wasm/action.yaml +++ b/.github/actions/build-policy-wasm/action.yaml @@ -8,6 +8,33 @@ description: > runs: using: composite steps: + # build_policy_wasm.sh already prefers a cached binary at .cache/opa// before reaching + # out to openpolicyagent.org, but nothing ever populated that directory in CI, so every run + # depended on that download — and it has repeatedly failed, taking a required job down with it. + # Restoring the cache means the fetch is only needed the first time a pinned version is seen. + - name: Resolve pinned OPA version + id: opa + shell: bash + env: + REPO_ROOT: ${{ github.action_path }}/../../.. + run: | + # Parse rather than source: the script builds the WASM on execution, so sourcing it here + # would do the work twice (and before the cache is restored). + version="$(sed -n 's/^OPA_VERSION="\${OPA_VERSION:-\([^}]*\)}"/\1/p' "${REPO_ROOT}/script/build_policy_wasm.sh" | head -1)" + if [ -z "${version}" ]; then + echo "Could not parse the pinned OPA version from script/build_policy_wasm.sh" >&2 + echo "If the OPA_VERSION line moved, update this step — a wrong cache key silently disables caching." >&2 + exit 1 + fi + echo "version=${version}" >> "${GITHUB_OUTPUT}" + echo "Pinned OPA version: ${version}" + + - name: Cache the pinned OPA binary + uses: actions/cache@v4 + with: + path: .cache/opa + key: opa-${{ runner.os }}-${{ runner.arch }}-${{ steps.opa.outputs.version }} + - name: Build policy WASM shell: bash env: diff --git a/script/build_policy_wasm.sh b/script/build_policy_wasm.sh index 90abf84943a..c924635d7a3 100755 --- a/script/build_policy_wasm.sh +++ b/script/build_policy_wasm.sh @@ -112,12 +112,12 @@ download_opa() { url="${OPA_DOWNLOAD_BASE_URL}/${OPA_VERSION}/${asset}" sha_url="${url}.sha256" echo "Downloading OPA ${OPA_VERSION} from ${url}..." >&2 - if ! curl -fsSL "${url}" -o "${tmp_bin}"; then + if ! curl -fsSL --retry 3 --retry-delay 2 --retry-all-errors "${url}" -o "${tmp_bin}"; then echo "Failed to download OPA binary from ${url}." >&2 print_opa_help "${asset}" exit 1 fi - if ! curl -fsSL "${sha_url}" -o "${tmp_sha}"; then + if ! curl -fsSL --retry 3 --retry-delay 2 --retry-all-errors "${sha_url}" -o "${tmp_sha}"; then echo "Failed to download OPA checksum from ${sha_url}." >&2 print_opa_help "${asset}" exit 1 From 9ce6f6df675b4268f75d4ec18920bbd1912a7f61 Mon Sep 17 00:00:00 2001 From: Sandy Chapman Date: Wed, 29 Jul 2026 13:35:19 -0300 Subject: [PATCH 2/3] ci: bound each OPA download attempt with connect and transfer timeouts The retries added alongside the cache were only half a fix: with no --max-time, a stalled connection consumes the entire job and the retries never get a turn. The failure that motivated this work burned 5m40s before giving up, which is consistent with exactly that. Both curl calls now set --connect-timeout 15 with --max-time 120 for the binary (~25x headroom for ~23 MB on a CI runner) and --max-time 60 for the much smaller checksum. Retry, output and error handling are unchanged. Validated: cold cache still downloads and populates .cache/opa//; warm cache still skips the network entirely; and against a server that accepts then never responds, each attempt is now cut short and retried (curl exit 28) instead of hanging. Signed-off-by: Sandy Chapman --- script/build_policy_wasm.sh | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/script/build_policy_wasm.sh b/script/build_policy_wasm.sh index c924635d7a3..417e145e2c5 100755 --- a/script/build_policy_wasm.sh +++ b/script/build_policy_wasm.sh @@ -112,12 +112,16 @@ download_opa() { url="${OPA_DOWNLOAD_BASE_URL}/${OPA_VERSION}/${asset}" sha_url="${url}.sha256" echo "Downloading OPA ${OPA_VERSION} from ${url}..." >&2 - if ! curl -fsSL --retry 3 --retry-delay 2 --retry-all-errors "${url}" -o "${tmp_bin}"; then + # Bound each attempt: without --max-time a stalled connection consumes the whole job and the + # retries never get a turn. 120s is ~25x headroom for a ~23 MB binary on a CI runner. + if ! curl -fsSL --retry 3 --retry-delay 2 --retry-all-errors \ + --connect-timeout 15 --max-time 120 "${url}" -o "${tmp_bin}"; then echo "Failed to download OPA binary from ${url}." >&2 print_opa_help "${asset}" exit 1 fi - if ! curl -fsSL --retry 3 --retry-delay 2 --retry-all-errors "${sha_url}" -o "${tmp_sha}"; then + if ! curl -fsSL --retry 3 --retry-delay 2 --retry-all-errors \ + --connect-timeout 15 --max-time 60 "${sha_url}" -o "${tmp_sha}"; then echo "Failed to download OPA checksum from ${sha_url}." >&2 print_opa_help "${asset}" exit 1 From ecc5ab74a95511eaea2fef1bd33a4683109a59fc Mon Sep 17 00:00:00 2001 From: Sandy Chapman Date: Wed, 29 Jul 2026 14:06:34 -0300 Subject: [PATCH 3/3] ci: cap the aggregate OPA retry window, not just each attempt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --max-time bounds one transfer and resets on every retry, so adding it alongside --retry 3 left the aggregate unbounded: worst case 4x120s + delays = 486s for the binary and 246s for the checksum, 732s combined. That collides with callers that impose their own ceiling. embedded_pdp/policy_wasm.py runs this script under DEFAULT_BUILD_TIMEOUT_SECONDS=120, so the subprocess would be killed mid-retry — the retries could never even finish, which defeats the point of adding them. Both calls now also set --retry-max-time, and the per-attempt budgets are sized to fit: binary <=45s aggregate + <=30s final attempt, checksum <=15s + <=10s, so worst case is ~100s and the `opa build` still fits inside 120s. The coupling to that caller is spelled out in a comment so the numbers are not re-tuned in isolation. Validated: cold download still succeeds (7s) and populates the cache; warm cache still skips the network; and against a server that accepts then never responds, the aggregate is cut at 62s instead of the ~126s the previous flags would have allowed. Signed-off-by: Sandy Chapman --- script/build_policy_wasm.sh | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/script/build_policy_wasm.sh b/script/build_policy_wasm.sh index 417e145e2c5..dd93be78e59 100755 --- a/script/build_policy_wasm.sh +++ b/script/build_policy_wasm.sh @@ -112,16 +112,20 @@ download_opa() { url="${OPA_DOWNLOAD_BASE_URL}/${OPA_VERSION}/${asset}" sha_url="${url}.sha256" echo "Downloading OPA ${OPA_VERSION} from ${url}..." >&2 - # Bound each attempt: without --max-time a stalled connection consumes the whole job and the - # retries never get a turn. 120s is ~25x headroom for a ~23 MB binary on a CI runner. + # Bound both a single attempt (--max-time) and the whole retry window (--retry-max-time). + # --max-time RESETS on each retry, so without --retry-max-time the aggregate is unbounded: 4 + # attempts could run for minutes. That matters because callers impose their own ceiling — + # embedded_pdp/policy_wasm.py runs this script with DEFAULT_BUILD_TIMEOUT_SECONDS=120 — and would + # kill the build mid-retry. Budget: <=45s aggregate + <=30s final attempt here, <=15s + <=10s for + # the checksum, leaving headroom for the `opa build` itself inside 120s. if ! curl -fsSL --retry 3 --retry-delay 2 --retry-all-errors \ - --connect-timeout 15 --max-time 120 "${url}" -o "${tmp_bin}"; then + --connect-timeout 10 --max-time 30 --retry-max-time 45 "${url}" -o "${tmp_bin}"; then echo "Failed to download OPA binary from ${url}." >&2 print_opa_help "${asset}" exit 1 fi if ! curl -fsSL --retry 3 --retry-delay 2 --retry-all-errors \ - --connect-timeout 15 --max-time 60 "${sha_url}" -o "${tmp_sha}"; then + --connect-timeout 10 --max-time 10 --retry-max-time 15 "${sha_url}" -o "${tmp_sha}"; then echo "Failed to download OPA checksum from ${sha_url}." >&2 print_opa_help "${asset}" exit 1