fix(OMN-13687): bounded retry on docker push to ECR — absorb transient runner→ECR EOF (emergency prod recovery)

[jonahgabriel]

Context

EMERGENCY PROD RECOVERY — OMN-13687 (operator-authorized).

The runtime image build (build-and-push-runtime.yml) builds and Trivy-scans cleanly on main HEAD 2cd720817 (the OMN-13670 crypto-floor hotfix #2131), but the final docker push to ECR repeatedly fails with a transient network EOF on a single blob HEAD request from the .201 self-hosted runner fleet to 272493677981.dkr.ecr.us-east-1.amazonaws.com. Two fresh workflow_dispatch runs on main today (runs 28315044412, 28315257828) both pushed several blobs successfully, then died on a blob HEAD with:

failed to do request: Head ".../v2/omninode-runtime/blobs/sha256:d08926bc...": EOF

Because the EOF recurs across different runs/runners, it is not one bad runner. No fresh omninode-runtime image has reached ECR since 2026-05-25 (sha256:de9a7a4c), which blocks the OMN-13418-gated prod re-pin.

Change

Wrap the docker push in build-and-push-runtime.yml in a bounded retry loop (5 attempts, exponential backoff 10s, 20s, 40s, 80s, cap 120s), mirroring the existing exponential-backoff resilience idiom already used by .github/actions/resolve-ecr-digest. docker push is resumable -- already-pushed blobs are skipped on retry -- so retrying absorbs the transient blob-HEAD EOF.

Additive only. The build, the image tag/digest computation, the lineage-guard, and the Trivy step are all unchanged. Only the network push is retried.

This PR does NOT deploy, restart, retag, or touch any cluster or .201 host config. The prod re-pin follows separately under OMN-13418 gating.

dod_evidence

Evidence-Ticket: OMN-13687
Evidence-Source: OCC#3241
Evidence-Class: hotfix
Active-Hotfix-PR: this PR

hotfix-evidence: OCC-3241
backmerge: #2131

Local gate results (2026-06-28):

• pre-commit run --files .github/workflows/build-and-push-runtime.yml -- all applicable hooks Passed (yamlfmt, skip-token rejector, root-cleanliness, etc.)
• yaml.safe_load(...) on the workflow -- YAML valid
• Diff: +35 / -1, scoped entirely to the Push image to Amazon ECR step

Backmerge note

The workflow retry loop should also land on dev to prevent re-divergence on the next dev to main promotion; tracked under OMN-13687. backmerge: #2131 references the prior emergency-recovery hotfix in the same OMN-13670/OMN-13687 recovery chain (same pattern #2131 used referencing #2128).

check-refresh: 2026-06-28T08:16:54Z

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 48503350-3701-4bfd-87e6-8b32183bd8fd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch hotfix/omn-13687-ecr-push-retry

---

_{Comment @coderabbitai help to get the list of available commands.}