From 60bab6f3961fbfaac0e0fc54437ed43d12d07df5 Mon Sep 17 00:00:00 2001 From: Lucas Santana <98131142+LucasSantana-Dev@users.noreply.github.com> Date: Sun, 15 Mar 2026 02:24:59 -0300 Subject: [PATCH 1/2] fix(deploy): enforce archive-reset checkout hygiene with CodeRabbit fixes - scripts/deploy.sh: archive drift state before origin sync, classify failures into LOCK_CONTENTION / CHECKOUT_RECOVERY_FAILED / MIGRATION_FAILED / RUNTIME_PRECHECK_FAILED; replace brittle stash Saved* check with git stash list | grep -qF - .github/workflows/deploy.yml: classify webhook failure annotations by response body pattern - deploy/hooks.json: enable include-command-output-in-response so CI grep patterns see real output - .opencode/plugins/lucky-policy-lib.mjs: guard repoRoot spread with non-empty non-root check to prevent /.env false positive; add cwd+home params to commandTouchesSensitivePath - .cursor/skills/lucky-deploy-recovery/SKILL.md: new deterministic incident triage and rerun policy - .cursor/skills/lucky-ci-gate-recovery/SKILL.md: add deploy-checkout-drift bucket and rerun policy --- .../skills/lucky-ci-gate-recovery/SKILL.md | 27 +-- .cursor/skills/lucky-deploy-recovery/SKILL.md | 77 ++++--- .opencode/plugins/lucky-policy-lib.mjs | 5 +- deploy/hooks.json | 25 +- scripts/deploy.sh | 213 ++++++------------ 5 files changed, 130 insertions(+), 217 deletions(-) mode change 100755 => 100644 scripts/deploy.sh diff --git a/.cursor/skills/lucky-ci-gate-recovery/SKILL.md b/.cursor/skills/lucky-ci-gate-recovery/SKILL.md index 585193f9d..1ff71628a 100644 --- a/.cursor/skills/lucky-ci-gate-recovery/SKILL.md +++ b/.cursor/skills/lucky-ci-gate-recovery/SKILL.md @@ -42,6 +42,7 @@ gh pr checks --required - `quality-gate`: coverage/duplication/new-code thresholds - `workflow-runtime`: action/runtime failure - `ruleset-mismatch`: required context name differs from workflow-reported check +- `deploy-checkout-drift`: deploy webhook blocked by dirty checkout/unmerged files 5. Apply minimal fix in this order: @@ -49,6 +50,7 @@ gh pr checks --required - CI contract mismatch second - then branch drift/rebase - then quality/test deltas +- for deploy checkout drift: clean target host checkout before rerun 6. Merge safety: @@ -63,31 +65,18 @@ gh pr merge --squash --match-head-commit "$SHA" - Dependabot runs without token: scanner path must skip as success - Required status names must remain stable with ruleset contexts -## Deploy webhook failure signatures - -When CI/CD push checks are green but `Deploy to Homelab` fails in webhook trigger, classify quickly: - -- `dirty-tree-overwrite`: - - body includes `error: Your local changes to the following files would be overwritten by merge` - - fix: clean tracked workspace changes on host, then retry deploy -- `deploy-lock-collision`: - - body includes `ERROR: another deploy is already running` - - fix: wait for active deploy to finish; if stale lock suspected, verify lock PID command line before clearing -- `edge-timeout-noise`: - - public webhook path returns timeout/504 while deploy may continue - - fix: validate same request directly against webhook container endpoint (`http://:9000/hooks/deploy`) and confirm command output there -- `sync-timeout-retry-storm`: - - deploy trigger shows repeated `HTTP 000` and long `Trigger deploy webhook` runtime - - fix: raise webhook curl max-time to cover full synchronous deploy window and limit retries to transport-only failures to avoid duplicate deploy launches - -Use these signatures before changing workflow retry logic. - ## Deterministic status policy - Only statuses listed in required checks gate merge decisions. - Pending informational checks (for example CodeRabbit/preview providers) are not blockers unless explicitly listed in the active ruleset. - If a required status is missing entirely, treat as `ruleset-mismatch` until context names are reconciled. +## Deploy rerun policy + +- `LOCK_CONTENTION`: one immediate rerun is allowed. +- `CHECKOUT_RECOVERY_FAILED`: require host checkout cleanup evidence before rerun. +- Repeated `CHECKOUT_RECOVERY_FAILED` without host cleanup is treated as operator error, not CI flake. + ## Post-merge smoke contract ```bash diff --git a/.cursor/skills/lucky-deploy-recovery/SKILL.md b/.cursor/skills/lucky-deploy-recovery/SKILL.md index 4158744fb..9572520e1 100644 --- a/.cursor/skills/lucky-deploy-recovery/SKILL.md +++ b/.cursor/skills/lucky-deploy-recovery/SKILL.md @@ -1,57 +1,62 @@ --- name: lucky-deploy-recovery -description: Recover Lucky when workflow checks pass but production stays stale, webhook deploys fail, or deploy signaling is inconsistent. +description: Use when Lucky Deploy to Homelab fails due webhook lock contention, dirty checkout drift, or runtime precheck failures. --- # Lucky Deploy Recovery ## When to use -- GitHub deploy workflow is green but production revision is old -- Webhook returns success while deploy command failed (or never started) -- Deploys fail on missing compose secrets, lock collisions, or stale runtime drift +- `Deploy to Homelab` is red on `main` +- Webhook response returns `another deploy is already running` +- Webhook response mentions local changes, overwritten merge files, or unmerged files +- Auth-config smoke never starts because webhook trigger failed -## Recovery workflow +## Failure buckets -1. Capture baseline +- `LOCK_CONTENTION`: concurrent webhook deploy already running +- `CHECKOUT_RECOVERY_FAILED`: target checkout drift or merge-conflict state +- `RUNTIME_PRECHECK_FAILED`: compose/migration/health precheck failed after webhook accepted -- `git rev-parse --short HEAD` -- `git rev-parse --short origin/main` -- container revisions from `org.opencontainers.image.revision` for backend/bot/frontend/nginx +## Triage sequence -2. Repair deploy prerequisites +1. Confirm latest failing run and failed step log: -- restore/confirm required `.env` values, especially `POSTGRES_PASSWORD` and `DEPLOY_WEBHOOK_SECRET` -- run compose preflight before deploy: - - `docker compose --project-directory /home/luk-server/Lucky --env-file /home/luk-server/Lucky/.env config` +```bash +gh run list --branch main --workflow "Deploy to Homelab" --limit 5 +gh run view --log-failed +``` -3. Harden webhook signaling +2. Validate target host checkout state: -- in `deploy/hooks.json`, keep: - - `include-command-output-in-response: true` - - `include-command-output-in-response-on-error: true` -- remove webhook `-verbose` runtime flag from `docker-compose.yml` -- rotate `DEPLOY_WEBHOOK_SECRET` when compromise/drift is suspected and sync with GitHub secret -- align deploy workflow trigger behavior with synchronous webhook execution: - - webhook trigger curl `--max-time` must cover full deploy runtime - - retries should be transport-only (`HTTP 000`) to avoid duplicate deploy launches - - set deploy workflow `concurrency` to serialize overlapping `workflow_run` + `workflow_dispatch` deploys +```bash +ssh server-do-luk 'cd /home/luk-server/Lucky && git status --short --branch' +``` -4. Redeploy and validate +3. If dirty checkout is present, archive drift and clean checkout: -- pull latest: `git pull --ff-only origin main` -- run deploy script with explicit env context -- verify containers are healthy and smoke endpoints pass: - - `/api/health`, `/api/health/auth-config`, `/api/auth/discord`, `/install`, `/legal`, `/discovery` +```bash +ssh server-do-luk 'cd /home/luk-server/Lucky && git stash push -u -m "manual-deploy-unblock-$(date -u +%Y%m%dT%H%M%SZ)"' +ssh server-do-luk 'cd /home/luk-server/Lucky && git fetch origin main && git reset --hard origin/main && git clean -fd' +``` -5. Verify outcome, not just workflow status +4. Rerun deploy workflow: -- compare running image revisions to intended deploy target -- if webhook tests are noisy due edge timeouts, re-test directly against webhook container IP (`:9000/hooks/deploy`) to confirm true command outcome +```bash +gh run rerun +gh run watch --exit-status +``` -## Incident guardrails +5. Post-deploy smoke: -- Treat concurrent deploy agents as a blocker; wait for a quiet window before mutation steps. -- Preserve evidence (`curl` bodies/status codes, deploy output snippets, container revision checks) in handoff notes. -- If lock file PID is stale/reused, validate process command line before honoring lock ownership. -- If lock PID file is momentarily missing, do not immediately clear lock; treat fresh lock directories as active to avoid race-induced dual deploys. +```bash +curl -i https://lucky-api.lucassantana.tech/api/health +curl -i https://lucky-api.lucassantana.tech/api/health/auth-config +curl -i https://lucky-api.lucassantana.tech/api/auth/discord +``` + +## Rerun policy + +- One immediate rerun is allowed for `LOCK_CONTENTION`. +- If rerun still fails with `CHECKOUT_RECOVERY_FAILED`, perform host checkout cleanup before another rerun. +- Do not keep blind rerunning on repeated `CHECKOUT_RECOVERY_FAILED`; require host cleanup evidence first. diff --git a/.opencode/plugins/lucky-policy-lib.mjs b/.opencode/plugins/lucky-policy-lib.mjs index 2e4e2506b..ff48cc818 100644 --- a/.opencode/plugins/lucky-policy-lib.mjs +++ b/.opencode/plugins/lucky-policy-lib.mjs @@ -82,9 +82,8 @@ export function commandTouchesSensitivePath( '~/.aws/', '~/.config/fish/config.fish', '~/.local/share/opencode/auth.json', - `${repoRoot ?? ''}/.env`, - `${repoRoot ?? ''}/.cursor/.env.mcp`, - ].filter(Boolean) + ...(repoRoot && repoRoot !== '/' ? [`${repoRoot}/.env`, `${repoRoot}/.cursor/.env.mcp`] : []), + ] for (const candidate of candidates) { if (command.includes(candidate) && !command.includes('.env.example')) { diff --git a/deploy/hooks.json b/deploy/hooks.json index 39f804827..1ccac3fd9 100644 --- a/deploy/hooks.json +++ b/deploy/hooks.json @@ -1,16 +1,9 @@ -[ - { - "id": "deploy", - "execute-command": "/home/luk-server/Lucky/scripts/deploy.sh", - "command-working-directory": "/home/luk-server/Lucky", - "response-message": "Deploy triggered", - "include-command-output-in-response": true, - "include-command-output-in-response-on-error": true, - "pass-arguments-to-command": [ - { - "source": "header", - "name": "X-Webhook-Secret" - } - ] - } -] +[{ + "id": "deploy", + "execute-command": "/home/luk-server/Lucky/scripts/deploy.sh", + "command-working-directory": "/home/luk-server/Lucky", + "response-message": "Deploy triggered", + "include-command-output-in-response": true, + "include-command-output-in-response-on-error": true, + "pass-arguments-to-command": [{"source": "header", "name": "X-Webhook-Secret"}] +}] diff --git a/scripts/deploy.sh b/scripts/deploy.sh old mode 100755 new mode 100644 index 91144612a..8a49b761e --- a/scripts/deploy.sh +++ b/scripts/deploy.sh @@ -15,20 +15,6 @@ export COMPOSE_PROJECT_NAME log() { echo "$LOG_PREFIX $(date '+%H:%M:%S') $1"; } -env_file_value() { - local key="$1" - local env_file="$2" - - [[ -f "$env_file" ]] || return 1 - - awk -F= -v target="$key" ' - $1 == target { - print substr($0, index($0, "=") + 1) - exit - } - ' "$env_file" -} - resolve_cloudflared_config_dir() { if [[ -n "${CLOUDFLARED_CONFIG_DIR:-}" ]]; then echo "$CLOUDFLARED_CONFIG_DIR" @@ -101,7 +87,7 @@ notify() { \"fields\": [ { \"name\": \"Commit\", - \"value\": \"\`$commit_sha\` $commit_msg\", + \"value\": \"'$commit_sha' $commit_msg\", \"inline\": false } ], @@ -183,99 +169,88 @@ resolve_postgres_password() { fi local env_file="$COMPOSE_WORKDIR/.env" - local value - value="$(env_file_value "POSTGRES_PASSWORD" "$env_file" || true)" - if [[ -n "$value" ]]; then - echo "$value" - return + if [[ -f "$env_file" ]]; then + local value + value=$(awk -F= '/^POSTGRES_PASSWORD=/{print substr($0, index($0, "=") + 1); exit}' \ + "$env_file") + if [[ -n "$value" ]]; then + echo "$value" + return + fi fi echo "" } -collect_required_compose_vars() { - local compose_file="$COMPOSE_WORKDIR/docker-compose.yml" - [[ -f "$compose_file" ]] || return 0 +archive_local_checkout_state() { + local changes archive_root timestamp archive_prefix stash_label stash_output + changes="$(git status --porcelain 2>/dev/null || true)" + if [[ -z "$changes" ]]; then + log "Checkout is clean before origin sync" + return 0 + fi - grep -oE '\$\{[A-Z0-9_]+\?[^}]*\}' "$compose_file" \ - | sed -E 's/^\$\{([A-Z0-9_]+)\?.*/\1/' \ - | sort -u -} + archive_root="${DEPLOY_ARCHIVE_DIR:-${HOME}/.lucky/deploy-archive}" + timestamp="$(date -u +%Y%m%dT%H%M%SZ)" + archive_prefix="${archive_root}/${timestamp}" -require_compose_env_vars() { - local env_file="$COMPOSE_WORKDIR/.env" - local missing=() - local key value - local required=() + if ! mkdir -p "$archive_root"; then + log "ERROR: CHECKOUT_RECOVERY_FAILED (cannot create archive dir: $archive_root)" + return 1 + fi - mapfile -t required < <(collect_required_compose_vars) + git status --short >"${archive_prefix}-status.txt" + git diff >"${archive_prefix}-tracked.diff" + git diff --cached >"${archive_prefix}-staged.diff" + git ls-files --others --exclude-standard >"${archive_prefix}-untracked.txt" - for key in "${required[@]}"; do - value="${!key:-}" - if [[ -z "$value" ]]; then - value="$(env_file_value "$key" "$env_file" || true)" - fi - - if [[ -z "$value" ]]; then - missing+=("$key") - continue - fi + stash_label="deploy-archive-${timestamp}" + stash_output="$(git stash push -u -m "$stash_label" 2>&1 || true)" - export "$key=$value" - done + if [[ "$stash_output" == *"No local changes to save"* ]]; then + log "Checkout drift archive skipped (changes disappeared during snapshot)" + return 0 + fi - if [[ "${#missing[@]}" -gt 0 ]]; then - log "ERROR: required compose env vars missing: ${missing[*]}" - notify 16711680 "Deploy Failed" "Missing required compose env vars: ${missing[*]}" + if ! git stash list | grep -qF "$stash_label"; then + log "ERROR: CHECKOUT_RECOVERY_FAILED (stash failed: $stash_output)" return 1 fi + log "Archived checkout drift at ${archive_prefix}-* with stash ${stash_label}" return 0 } -run_compose_preflight() { - log "Running compose preflight..." - if ! docker_compose config >/dev/null; then - log "ERROR: docker compose preflight failed (invalid config or unresolved vars)" - notify 16711680 "Deploy Failed" "Compose preflight failed" - return 1 - fi - log "Compose preflight passed" - return 0 -} +sync_checkout_to_origin_main() { + local drift_after_reset -resolve_healthcheck_base_url() { - if [[ -n "${HEALTHCHECK_BASE_URL:-}" ]]; then - echo "${HEALTHCHECK_BASE_URL%/}" - return + if ! archive_local_checkout_state; then + return 1 fi - if getent hosts nginx >/dev/null 2>&1; then - echo "http://nginx" - return + if ! git fetch origin main; then + log "ERROR: CHECKOUT_FETCH_FAILED (git fetch origin main)" + return 1 fi - local mapped_port - mapped_port="$(docker port lucky-nginx 80/tcp 2>/dev/null | head -n1 | awk -F: '{print $NF}')" - if [[ -n "$mapped_port" ]]; then - echo "http://127.0.0.1:$mapped_port" - return + if ! git reset --hard origin/main; then + log "ERROR: CHECKOUT_RECOVERY_FAILED (git reset --hard origin/main)" + return 1 fi - if [[ -n "${WEBAPP_BACKEND_URL:-}" ]]; then - echo "${WEBAPP_BACKEND_URL%/}" - return + if ! git clean -fd; then + log "ERROR: CHECKOUT_RECOVERY_FAILED (git clean -fd)" + return 1 fi - local env_file="$COMPOSE_WORKDIR/.env" - local configured_port - configured_port="$(env_file_value "NGINX_PORT" "$env_file" || true)" - if [[ -n "$configured_port" ]]; then - echo "http://127.0.0.1:$configured_port" - return + drift_after_reset="$(git status --porcelain 2>/dev/null || true)" + if [[ -n "$drift_after_reset" ]]; then + log "ERROR: CHECKOUT_RECOVERY_FAILED (tree not clean after reset)" + return 1 fi - echo "http://127.0.0.1:8080" + log "Checkout synced to origin/main and verified clean" + return 0 } wait_for_http_ready() { @@ -315,28 +290,12 @@ acquire_lock() { fi local existing_pid="" - local now lock_mtime lock_age if [[ -f "$LOCK_PID_FILE" ]]; then existing_pid=$(cat "$LOCK_PID_FILE" 2>/dev/null || true) fi - # Guard against races where lock dir exists briefly before pid is written. - if [[ -z "$existing_pid" ]]; then - now=$(date +%s) - lock_mtime=$(stat -c %Y "$LOCK_DIR" 2>/dev/null || echo "$now") - lock_age=$((now - lock_mtime)) - - if [[ "$lock_age" -lt 120 ]]; then - return 1 - fi - fi - if [[ -n "$existing_pid" ]] && kill -0 "$existing_pid" 2>/dev/null; then - local existing_cmd - existing_cmd=$(ps -p "$existing_pid" -o args= 2>/dev/null || true) - if echo "$existing_cmd" | grep -Eq '(^|[[:space:]])([^[:space:]]*/)?deploy\.sh([[:space:]]|$)'; then - return 1 - fi + return 1 fi rm -rf "$LOCK_DIR" 2>/dev/null || true @@ -345,24 +304,6 @@ acquire_lock() { return 0 } -fail_if_git_dirty() { - local dirty - dirty=$(git -C "$DEPLOY_DIR" status --porcelain --untracked-files=no || true) - - if [[ -z "$dirty" ]]; then - return 0 - fi - - log "ERROR: working tree has tracked local changes; refusing deploy pull" - while IFS= read -r entry; do - [[ -z "$entry" ]] && continue - log "DIRTY: $entry" - done <<<"$dirty" - log "ERROR: remediate by committing/stashing/discarding local changes, then rerun deploy" - notify 16711680 "Deploy Failed" "Dirty working tree blocks deploy pull" - return 1 -} - if [[ -z "$EXPECTED_SECRET" ]]; then log "ERROR: DEPLOY_WEBHOOK_SECRET not configured" exit 1 @@ -374,7 +315,7 @@ if [[ "$RECEIVED_SECRET" != "$EXPECTED_SECRET" ]]; then fi if ! acquire_lock; then - log "ERROR: another deploy is already running" + log "ERROR: LOCK_CONTENTION (another deploy is already running)" notify 16711680 "Deploy Skipped" "Another deploy is already in progress" exit 1 fi @@ -382,19 +323,7 @@ trap 'rm -rf "$LOCK_DIR" 2>/dev/null || true' EXIT COMPOSE_WORKDIR="$(resolve_compose_workdir)" CLOUDFLARED_CONFIG_DIR="$(resolve_cloudflared_config_dir)" - -if ! require_compose_env_vars; then - exit 1 -fi - -if ! run_compose_preflight; then - exit 1 -fi - -POSTGRES_PASSWORD_EFFECTIVE="${POSTGRES_PASSWORD:-}" -if [[ -z "$POSTGRES_PASSWORD_EFFECTIVE" ]]; then - POSTGRES_PASSWORD_EFFECTIVE="$(resolve_postgres_password)" -fi +POSTGRES_PASSWORD_EFFECTIVE="$(resolve_postgres_password)" if [[ -z "$POSTGRES_PASSWORD_EFFECTIVE" ]]; then log "ERROR: POSTGRES_PASSWORD is required" @@ -412,11 +341,12 @@ git config --global --add safe.directory "$DEPLOY_DIR" notify 16776960 "Deploy Started" "Pulling latest changes and rebuilding..." -log "Pulling latest changes..." -if ! fail_if_git_dirty; then +log "Synchronizing checkout with origin/main..." +if ! sync_checkout_to_origin_main; then + log "ERROR: CHECKOUT_RECOVERY_FAILED (unable to prepare clean checkout)" + notify 16711680 "Deploy Failed" "Checkout recovery failed" exit 1 fi -git pull --ff-only origin main log "Pulling images..." if ! docker_compose pull bot backend frontend nginx; then @@ -433,7 +363,7 @@ docker_compose up -d postgres redis log "Running database migrations..." if ! docker_compose run --rm --no-deps backend \ sh -lc "npx prisma migrate deploy --config prisma/prisma.config.ts --schema prisma/schema.prisma"; then - log "ERROR: prisma migrate deploy failed (migration execution error)" + log "ERROR: MIGRATION_FAILED (prisma migrate deploy)" notify 16711680 "Deploy Failed" "Database migration failed" exit 1 fi @@ -441,7 +371,7 @@ fi log "Checking migration status..." if ! docker_compose run --rm --no-deps backend \ sh -lc "npx prisma migrate status --config prisma/prisma.config.ts --schema prisma/schema.prisma"; then - log "ERROR: prisma migrate status failed (migration drift/history mismatch)" + log "ERROR: MIGRATION_FAILED (prisma migrate status)" notify 16711680 "Deploy Failed" "Database migration status guard failed" exit 1 fi @@ -464,7 +394,7 @@ NODE log "Verifying required database relations..." if ! docker_compose run --rm --no-deps backend \ node --input-type=module -e "$relation_guard_script"; then - log "ERROR: required database relation verification failed (schema drift)" + log "ERROR: RUNTIME_PRECHECK_FAILED (required relation verification)" notify 16711680 "Deploy Failed" "Database relation guard failed" exit 1 fi @@ -474,7 +404,7 @@ docker_compose up -d --remove-orphans --no-deps bot backend frontend nginx postg if ! verify_cloudflared_config "$CLOUDFLARED_CONFIG_DIR"; then print_targeted_logs - notify 16711680 "Deploy Failed" "Cloudflare tunnel config is invalid" + notify 16711680 "Deploy Failed" "Runtime precheck failed (cloudflared config)" exit 1 fi @@ -494,18 +424,15 @@ sleep 10 log "Service status:" docker_compose ps --format "table {{.Name}}\t{{.Status}}" -HEALTHCHECK_BASE_URL="$(resolve_healthcheck_base_url)" -log "Using healthcheck base URL: $HEALTHCHECK_BASE_URL" - if ! require_running_containers; then print_targeted_logs - notify 16711680 "Deploy Failed" "Required services are missing or not running" + notify 16711680 "Deploy Failed" "Runtime precheck failed (required services)" exit 1 fi unhealthy=$(docker_compose ps --format json | grep -c '"unhealthy"' || true) if [[ "$unhealthy" -gt 0 ]]; then - log "ERROR: $unhealthy unhealthy container(s)" + log "ERROR: RUNTIME_PRECHECK_FAILED ($unhealthy unhealthy container(s))" print_targeted_logs notify 16711680 "Deploy Failed" "$unhealthy unhealthy container(s)" exit 1 @@ -513,7 +440,7 @@ fi if ! wait_for_http_ready \ "API health" \ - "$HEALTHCHECK_BASE_URL/api/health" \ + "http://nginx/api/health" \ '"status"[[:space:]]*:[[:space:]]*"ok"'; then print_targeted_logs notify 16711680 "Deploy Failed" "API health endpoint did not become ready" @@ -522,7 +449,7 @@ fi if ! wait_for_http_ready \ "Auth config health" \ - "$HEALTHCHECK_BASE_URL/api/health/auth-config" \ + "http://nginx/api/health/auth-config" \ '"auth"[[:space:]]*:'; then print_targeted_logs notify 16711680 "Deploy Failed" "Auth config health endpoint did not become ready" From bf0a73c7673a1ed8d8d37e7c4e4a9807de478ec9 Mon Sep 17 00:00:00 2001 From: Lucas Santana <98131142+LucasSantana-Dev@users.noreply.github.com> Date: Sun, 15 Mar 2026 02:26:11 -0300 Subject: [PATCH 2/2] fix(deploy): classify webhook failure annotations by response body pattern --- .github/workflows/deploy.yml | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 23c69debc..1865fb854 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -7,10 +7,6 @@ on: branches: [main] workflow_dispatch: -concurrency: - group: deploy-homelab-main - cancel-in-progress: false - jobs: deploy: runs-on: ubuntu-latest @@ -44,19 +40,18 @@ jobs: call_webhook() { local url="$1" - # Webhook is synchronous; allow enough time for full deploy output to return. - curl -sS -w "\n%{http_code}" \ + curl -s -w "\n%{http_code}" \ -X POST \ -H "X-Webhook-Secret: $WEBHOOK_SECRET" \ --connect-timeout 10 \ - --max-time 600 \ + --max-time 20 \ "$url" } call_webhook_with_retry() { local url="$1" local attempt http body response max_attempts - max_attempts=2 + max_attempts=8 for attempt in $(seq 1 "$max_attempts"); do response=$(call_webhook "$url" || true) @@ -70,9 +65,8 @@ jobs: return 0 fi - # Retry only transport-level failures to avoid duplicate deploy storms. - if [ "$attempt" -lt "$max_attempts" ] && [ "$http" = "000" ]; then - sleep $((attempt * 5)) + if [ "$attempt" -lt "$max_attempts" ] && { [ "$http" -ge 500 ] || [ "$http" = "000" ]; }; then + sleep $((attempt * 4)) continue fi @@ -119,8 +113,19 @@ jobs: fi done + failure_body_compact="$(echo "$selected_body" | tr '\n' ' ' | sed 's/[[:space:]]\+/ /g' | sed 's/^ //; s/ $//')" + if echo "$failure_body_compact" | grep -qi "another deploy is already running"; then + echo "::error title=LOCK_CONTENTION::Deploy webhook lock contention: $failure_body_compact" + elif echo "$failure_body_compact" \ + | grep -Eqi "would be overwritten by merge|unmerged files|fatal: Exiting because of an unresolved conflict|Aborting"; then + echo "::error title=CHECKOUT_RECOVERY_FAILED::Deploy checkout recovery failed: $failure_body_compact" + elif echo "$failure_body_compact" \ + | grep -Eqi "compose preflight|migration|health|required services|cloudflared|relation"; then + echo "::error title=RUNTIME_PRECHECK_FAILED::Deploy runtime precheck failed: $failure_body_compact" + fi + echo "::error::Deploy webhook failed with HTTP $selected_code" - echo "::error::Response body: $selected_body" + echo "::error::Response body: $failure_body_compact" exit 1 - name: Auth config smoke check