fix(deploy): recover webhook rollouts from checkout drift

.cursor/skills/lucky-ci-gate-recovery/SKILL.md

@@ -42,13 +42,15 @@ gh pr checks <PR#> --required
 - `quality-gate`: coverage/duplication/new-code thresholds
 - `workflow-runtime`: action/runtime failure
 - `ruleset-mismatch`: required context name differs from workflow-reported check
+- `deploy-checkout-drift`: deploy webhook blocked by dirty checkout/unmerged files
 
 5. Apply minimal fix in this order:
 
 - Ruleset/context mismatch first
 - CI contract mismatch second
 - then branch drift/rebase
 - then quality/test deltas
+- for deploy checkout drift: clean target host checkout before rerun
 
 6. Merge safety:
 
@@ -63,31 +65,18 @@ gh pr merge <PR#> --squash --match-head-commit "$SHA"
 - Dependabot runs without token: scanner path must skip as success
 - Required status names must remain stable with ruleset contexts
 
-## Deploy webhook failure signatures
-
-When CI/CD push checks are green but `Deploy to Homelab` fails in webhook trigger, classify quickly:
-
-- `dirty-tree-overwrite`:
-    - body includes `error: Your local changes to the following files would be overwritten by merge`
-    - fix: clean tracked workspace changes on host, then retry deploy
-- `deploy-lock-collision`:
-    - body includes `ERROR: another deploy is already running`
-    - fix: wait for active deploy to finish; if stale lock suspected, verify lock PID command line before clearing
-- `edge-timeout-noise`:
-    - public webhook path returns timeout/504 while deploy may continue
-    - fix: validate same request directly against webhook container endpoint (`http://<webhook-ip>:9000/hooks/deploy`) and confirm command output there
-- `sync-timeout-retry-storm`:
-    - deploy trigger shows repeated `HTTP 000` and long `Trigger deploy webhook` runtime
-    - fix: raise webhook curl max-time to cover full synchronous deploy window and limit retries to transport-only failures to avoid duplicate deploy launches
-
-Use these signatures before changing workflow retry logic.
-
 ## Deterministic status policy
 
 - Only statuses listed in required checks gate merge decisions.
 - Pending informational checks (for example CodeRabbit/preview providers) are not blockers unless explicitly listed in the active ruleset.
 - If a required status is missing entirely, treat as `ruleset-mismatch` until context names are reconciled.
 
+## Deploy rerun policy
+
+- `LOCK_CONTENTION`: one immediate rerun is allowed.
+- `CHECKOUT_RECOVERY_FAILED`: require host checkout cleanup evidence before rerun.
+- Repeated `CHECKOUT_RECOVERY_FAILED` without host cleanup is treated as operator error, not CI flake.
+
 ## Post-merge smoke contract
 
 ```bash

.cursor/skills/lucky-deploy-recovery/SKILL.md

@@ -1,57 +1,62 @@
 ---
 name: lucky-deploy-recovery
-description: Recover Lucky when workflow checks pass but production stays stale, webhook deploys fail, or deploy signaling is inconsistent.
+description: Use when Lucky Deploy to Homelab fails due webhook lock contention, dirty checkout drift, or runtime precheck failures.
 ---
 
 # Lucky Deploy Recovery
 
 ## When to use
 
-- GitHub deploy workflow is green but production revision is old
-- Webhook returns success while deploy command failed (or never started)
-- Deploys fail on missing compose secrets, lock collisions, or stale runtime drift
+- `Deploy to Homelab` is red on `main`
+- Webhook response returns `another deploy is already running`
+- Webhook response mentions local changes, overwritten merge files, or unmerged files
+- Auth-config smoke never starts because webhook trigger failed
 
-## Recovery workflow
+## Failure buckets
 
-1. Capture baseline
+- `LOCK_CONTENTION`: concurrent webhook deploy already running
+- `CHECKOUT_RECOVERY_FAILED`: target checkout drift or merge-conflict state
+- `RUNTIME_PRECHECK_FAILED`: compose/migration/health precheck failed after webhook accepted
 
-- `git rev-parse --short HEAD`
-- `git rev-parse --short origin/main`
-- container revisions from `org.opencontainers.image.revision` for backend/bot/frontend/nginx
+## Triage sequence
 
-2. Repair deploy prerequisites
+1. Confirm latest failing run and failed step log:
 
-- restore/confirm required `.env` values, especially `POSTGRES_PASSWORD` and `DEPLOY_WEBHOOK_SECRET`
-- run compose preflight before deploy:
-    - `docker compose --project-directory /home/luk-server/Lucky --env-file /home/luk-server/Lucky/.env config`
+```bash
+gh run list --branch main --workflow "Deploy to Homelab" --limit 5
+gh run view <RUN_ID> --log-failed
+```
 
-3. Harden webhook signaling
+2. Validate target host checkout state:
 
-- in `deploy/hooks.json`, keep:
-    - `include-command-output-in-response: true`
-    - `include-command-output-in-response-on-error: true`
-- remove webhook `-verbose` runtime flag from `docker-compose.yml`
-- rotate `DEPLOY_WEBHOOK_SECRET` when compromise/drift is suspected and sync with GitHub secret
-- align deploy workflow trigger behavior with synchronous webhook execution:
-    - webhook trigger curl `--max-time` must cover full deploy runtime
-    - retries should be transport-only (`HTTP 000`) to avoid duplicate deploy launches
-    - set deploy workflow `concurrency` to serialize overlapping `workflow_run` + `workflow_dispatch` deploys
+```bash
+ssh server-do-luk 'cd /home/luk-server/Lucky && git status --short --branch'
+```
 
-4. Redeploy and validate
+3. If dirty checkout is present, archive drift and clean checkout:
 
-- pull latest: `git pull --ff-only origin main`
-- run deploy script with explicit env context
-- verify containers are healthy and smoke endpoints pass:
-    - `/api/health`, `/api/health/auth-config`, `/api/auth/discord`, `/install`, `/legal`, `/discovery`
+```bash
+ssh server-do-luk 'cd /home/luk-server/Lucky && git stash push -u -m "manual-deploy-unblock-$(date -u +%Y%m%dT%H%M%SZ)"'
+ssh server-do-luk 'cd /home/luk-server/Lucky && git fetch origin main && git reset --hard origin/main && git clean -fd'
+```
 
-5. Verify outcome, not just workflow status
+4. Rerun deploy workflow:
 
-- compare running image revisions to intended deploy target
-- if webhook tests are noisy due edge timeouts, re-test directly against webhook container IP (`:9000/hooks/deploy`) to confirm true command outcome
+```bash
+gh run rerun <RUN_ID>
+gh run watch <RUN_ID> --exit-status
+```
 
-## Incident guardrails
+5. Post-deploy smoke:
 
-- Treat concurrent deploy agents as a blocker; wait for a quiet window before mutation steps.
-- Preserve evidence (`curl` bodies/status codes, deploy output snippets, container revision checks) in handoff notes.
-- If lock file PID is stale/reused, validate process command line before honoring lock ownership.
-- If lock PID file is momentarily missing, do not immediately clear lock; treat fresh lock directories as active to avoid race-induced dual deploys.
+```bash
+curl -i https://lucky-api.lucassantana.tech/api/health
+curl -i https://lucky-api.lucassantana.tech/api/health/auth-config
+curl -i https://lucky-api.lucassantana.tech/api/auth/discord
+```
+
+## Rerun policy
+
+- One immediate rerun is allowed for `LOCK_CONTENTION`.
+- If rerun still fails with `CHECKOUT_RECOVERY_FAILED`, perform host checkout cleanup before another rerun.
+- Do not keep blind rerunning on repeated `CHECKOUT_RECOVERY_FAILED`; require host cleanup evidence first.

.github/workflows/deploy.yml

@@ -7,10 +7,6 @@ on:
         branches: [main]
     workflow_dispatch:
 
-concurrency:
-    group: deploy-homelab-main
-    cancel-in-progress: false
-
 jobs:
     deploy:
         runs-on: ubuntu-latest
@@ -44,19 +40,18 @@ jobs:
 
                   call_webhook() {
                     local url="$1"
-                    # Webhook is synchronous; allow enough time for full deploy output to return.
-                    curl -sS -w "\n%{http_code}" \
+                    curl -s -w "\n%{http_code}" \
                       -X POST \
                       -H "X-Webhook-Secret: $WEBHOOK_SECRET" \
                       --connect-timeout 10 \
-                      --max-time 600 \
+                      --max-time 20 \
                       "$url"
                   }
 
                   call_webhook_with_retry() {
                     local url="$1"
                     local attempt http body response max_attempts
-                    max_attempts=2
+                    max_attempts=8
 
                     for attempt in $(seq 1 "$max_attempts"); do
                       response=$(call_webhook "$url" || true)
@@ -70,9 +65,8 @@ jobs:
                         return 0
                       fi
 
-                      # Retry only transport-level failures to avoid duplicate deploy storms.
-                      if [ "$attempt" -lt "$max_attempts" ] && [ "$http" = "000" ]; then
-                        sleep $((attempt * 5))
+                      if [ "$attempt" -lt "$max_attempts" ] && { [ "$http" -ge 500 ] || [ "$http" = "000" ]; }; then
+                        sleep $((attempt * 4))
                         continue
                       fi
 
@@ -119,8 +113,19 @@ jobs:
                     fi
                   done
 
+                  failure_body_compact="$(echo "$selected_body" | tr '\n' ' ' | sed 's/[[:space:]]\+/ /g' | sed 's/^ //; s/ $//')"
+                  if echo "$failure_body_compact" | grep -qi "another deploy is already running"; then
+                    echo "::error title=LOCK_CONTENTION::Deploy webhook lock contention: $failure_body_compact"
+                  elif echo "$failure_body_compact" \
+                    | grep -Eqi "would be overwritten by merge|unmerged files|fatal: Exiting because of an unresolved conflict|Aborting"; then
+                    echo "::error title=CHECKOUT_RECOVERY_FAILED::Deploy checkout recovery failed: $failure_body_compact"
+                  elif echo "$failure_body_compact" \
+                    | grep -Eqi "compose preflight|migration|health|required services|cloudflared|relation"; then
+                    echo "::error title=RUNTIME_PRECHECK_FAILED::Deploy runtime precheck failed: $failure_body_compact"
+                  fi
+
                   echo "::error::Deploy webhook failed with HTTP $selected_code"
-                  echo "::error::Response body: $selected_body"
+                  echo "::error::Response body: $failure_body_compact"
                   exit 1
 
             - name: Auth config smoke check

.opencode/plugins/lucky-policy-lib.mjs

@@ -82,9 +82,8 @@ export function commandTouchesSensitivePath(
     '~/.aws/',
     '~/.config/fish/config.fish',
     '~/.local/share/opencode/auth.json',
-    `${repoRoot ?? ''}/.env`,
-    `${repoRoot ?? ''}/.cursor/.env.mcp`,
-  ].filter(Boolean)
+    ...(repoRoot && repoRoot !== '/' ? [`${repoRoot}/.env`, `${repoRoot}/.cursor/.env.mcp`] : []),
+  ]
 
   for (const candidate of candidates) {
     if (command.includes(candidate) && !command.includes('.env.example')) {

deploy/hooks.json

@@ -1,16 +1,9 @@
-[
-    {
-        "id": "deploy",
-        "execute-command": "/home/luk-server/Lucky/scripts/deploy.sh",
-        "command-working-directory": "/home/luk-server/Lucky",
-        "response-message": "Deploy triggered",
-        "include-command-output-in-response": true,
-        "include-command-output-in-response-on-error": true,
-        "pass-arguments-to-command": [
-            {
-                "source": "header",
-                "name": "X-Webhook-Secret"
-            }
-        ]
-    }
-]
+[{
+  "id": "deploy",
+  "execute-command": "/home/luk-server/Lucky/scripts/deploy.sh",
+  "command-working-directory": "/home/luk-server/Lucky",
+  "response-message": "Deploy triggered",
+  "include-command-output-in-response": true,
+  "include-command-output-in-response-on-error": true,
+  "pass-arguments-to-command": [{"source": "header", "name": "X-Webhook-Secret"}]
+}]