diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 4924bb435..d54365349 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -584,6 +584,50 @@ jobs: echo "::error::OAuth redirect smoke summary: non_rate_limited_attempts=$non_rate_limited_attempts, rate_limited_count=$rate_limited_count" exit 1 + - name: Wait for homelab deploy completion + if: steps.docker_wait.outputs.docker_rebuilt == 'true' + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + set -euo pipefail + if [ "${{ github.event_name }}" = "workflow_run" ]; then + expected="${{ github.event.workflow_run.head_sha }}" + else + expected="${{ github.sha }}" + fi + + echo "==> Waiting for homelab deploy to complete (SHA: $expected)..." + status_was_seen=false + + for attempt in $(seq 1 12); do + echo "Attempt ${attempt}/12..." + deploy_state=$(gh api \ + "repos/${{ github.repository }}/commits/${expected}/statuses" \ + --jq '[.[] | select(.context == "homelab-deploy")] | first | .state // ""' \ + 2>/dev/null || echo "") + + [ "$deploy_state" = "pending" ] && status_was_seen=true + + if [ "$deploy_state" = "success" ]; then + echo "==> Homelab deploy completed successfully." + exit 0 + fi + if [ "$deploy_state" = "failure" ] || [ "$deploy_state" = "error" ]; then + echo "::error::Homelab deploy reported failure after image deployment — check deploy logs on homelab" + exit 1 + fi + + if [ "$status_was_seen" = "false" ] && [ "$attempt" -eq 6 ]; then + echo "::warning::No homelab-deploy status after ~90s — GITHUB_DEPLOY_STATUS_TOKEN likely not configured on homelab. Cannot confirm full deploy completion." + exit 0 + fi + + echo "Deploy status: ${deploy_state:-none}. Waiting 15s..." + sleep 15 + done + + echo "::warning::Deploy completion timed out after 3 min — bot health gate may still be running. Check Discord notification for full outcome." + - name: Notify failure if: failure() run: echo "::error::Homelab deployment failed — check logs above" diff --git a/docs/decisions/2026-05-24-deploy-ci-deploy-completion-wait.md b/docs/decisions/2026-05-24-deploy-ci-deploy-completion-wait.md new file mode 100644 index 000000000..de8a40ca8 --- /dev/null +++ b/docs/decisions/2026-05-24-deploy-ci-deploy-completion-wait.md @@ -0,0 +1,86 @@ +# Deploy CI: wait for homelab deploy completion on `docker_rebuilt=true` path + +Date: 2026-05-24 + +## Context + +The Lucky CI deploy pipeline fires via `workflow_run` after Docker images are published. The +webhook handler runs `deploy.sh` asynchronously (`nohup ... & disown`), so the webhook +returns before `deploy.sh` completes. + +Gap B (PR #1046) added commit status posting to `deploy.sh`: it posts `pending` on sync, +then `success` or `failure` to the `homelab-deploy` GitHub commit status context on exit. + +CI's "Validate deployed version" step has two paths: + +- `docker_rebuilt=false` (no image rebuild): already polls `homelab-deploy` commit status, + waits for `success`/`failure`, and exits accordingly. +- `docker_rebuilt=true` (image rebuilt): polls `/api/health/version` for SHA match and exits + 0 on match — but does not wait for `deploy.sh` to complete. + +The race window: `/api/health/version` can return the new SHA as soon as the backend +container restarts. But `deploy.sh` continues executing (bot health poll, up to 90s). If +the bot health gate fails after CI sees the SHA match, CI marks the deploy step green while +`deploy.sh` exits non-zero and posts `failure` to `homelab-deploy`. Signals diverge. + +## Decision + +Add a "Wait for homelab deploy completion" step after all smoke checks, conditioned on +`docker_rebuilt=true`. The step polls the `homelab-deploy` commit status using `gh api` +for up to 3 minutes (12 × 15s). Outcomes: + +- `success` → CI passes. +- `failure`/`error` → CI fails with an explicit error. +- No status seen after 90s → warn and proceed (`GITHUB_DEPLOY_STATUS_TOKEN` not configured on + homelab; cannot confirm full completion — accepted degraded mode). +- Timeout (3 min) → warn and proceed (bot health gate is warn-on-timeout by design; Discord + notification carries the authoritative outcome). + +The `docker_rebuilt=false` path is unaffected — it already has equivalent logic in "Validate +deployed version". + +## Alternatives considered + +**Do nothing (accept + document)** — Bot health gate is warn-on-timeout by design; SHA match +and smoke checks are strong signals. Discord notification tells the operator about bot health. +Rejected: an operator reading "CI green" after a deploy with a divergent `homelab-deploy: +failure` status has to cross-check two systems to understand deploy outcome. Confusion is +real and bounded (~90s race window but repeatable). + +**Make webhook synchronous** — Remove `nohup ... & disown` from `deploy-wrapper.sh`; webhook +blocks until `deploy.sh` completes. True exit code reaches CI. Rejected: adds 3–5 min to +webhook response time; homelab nginx timeout risk; requires deploy-wrapper redesign. + +**Fail CI on missing `GITHUB_DEPLOY_STATUS_TOKEN`** — Hard-fail the step if no status appears +after 90s. Rejected: inconsistent with the `docker_rebuilt=false` path, which also falls +back gracefully when the token is absent. A missing token means the feature is disabled, not +that the deploy failed. + +## Consequences + +**Positive:** + +- CI outcome on `docker_rebuilt=true` now reflects the full deploy cycle, including bot health + gate, not just SHA publication. +- Consistent behavior between `docker_rebuilt=true` and `docker_rebuilt=false` paths: both + wait for the `homelab-deploy` commit status before the job completes. +- Adds at most ~90s to CI runtime (time from SHA match to bot gateway healthy). + +**Negative:** + +- Requires `GITHUB_DEPLOY_STATUS_TOKEN` to be configured on homelab. If absent, step warns + and proceeds — the race condition remains in degraded mode. +- If `deploy.sh` exit trap fails to call `post_deploy_status`, the step times out (warn). + +**Neutral:** + +- The bot health gate's warn-on-timeout design is preserved end-to-end: a Discord outage + causes `deploy.sh` to warn (not fail), so `homelab-deploy` still reaches `success`, and CI + follows. + +## Revisit when + +- `GITHUB_DEPLOY_STATUS_TOKEN` stability breaks on homelab → flip to Option 4 + enhanced + Discord alerting (post bot health outcome to channel explicitly). +- Bot health gate is changed from warn-on-timeout to fail-on-timeout → reassess whether + CI timeout warning is still appropriate.