LucasSantana-Dev · Mar 14, 2026 · Mar 14, 2026 · Mar 14, 2026
diff --git a/.cursor/skills/lucky-ci-gate-recovery/SKILL.md b/.cursor/skills/lucky-ci-gate-recovery/SKILL.md
@@ -63,6 +63,22 @@ gh pr merge <PR#> --squash --match-head-commit "$SHA"
 - Dependabot runs without token: scanner path must skip as success
 - Required status names must remain stable with ruleset contexts
 
+## Deploy webhook failure signatures
+
+When CI/CD push checks are green but `Deploy to Homelab` fails in webhook trigger, classify quickly:
+
+- `dirty-tree-overwrite`:
+    - body includes `error: Your local changes to the following files would be overwritten by merge`
+    - fix: clean tracked workspace changes on host, then retry deploy
+- `deploy-lock-collision`:
+    - body includes `ERROR: another deploy is already running`
+    - fix: wait for active deploy to finish; if stale lock suspected, verify lock PID command line before clearing
+- `edge-timeout-noise`:
+    - public webhook path returns timeout/504 while deploy may continue
+    - fix: validate same request directly against webhook container endpoint (`http://<webhook-ip>:9000/hooks/deploy`) and confirm command output there
+
+Use these signatures before changing workflow retry logic.
+
 ## Deterministic status policy
 
 - Only statuses listed in required checks gate merge decisions.

diff --git a/.cursor/skills/lucky-deploy-recovery/SKILL.md b/.cursor/skills/lucky-deploy-recovery/SKILL.md
@@ -0,0 +1,52 @@
+---
+name: lucky-deploy-recovery
+description: Recover Lucky when workflow checks pass but production stays stale, webhook deploys fail, or deploy signaling is inconsistent.
+---
+
+# Lucky Deploy Recovery
+
+## When to use
+
+- GitHub deploy workflow is green but production revision is old
+- Webhook returns success while deploy command failed (or never started)
+- Deploys fail on missing compose secrets, lock collisions, or stale runtime drift
+
+## Recovery workflow
+
+1. Capture baseline
+
+- `git rev-parse --short HEAD`
+- `git rev-parse --short origin/main`
+- container revisions from `org.opencontainers.image.revision` for backend/bot/frontend/nginx
+
+2. Repair deploy prerequisites
+
+- restore/confirm required `.env` values, especially `POSTGRES_PASSWORD` and `DEPLOY_WEBHOOK_SECRET`
+- run compose preflight before deploy:
+    - `docker compose --project-directory /home/luk-server/Lucky --env-file /home/luk-server/Lucky/.env config`
+
+3. Harden webhook signaling
+
+- in `deploy/hooks.json`, keep:
+    - `include-command-output-in-response: true`
+    - `include-command-output-in-response-on-error: true`
+- remove webhook `-verbose` runtime flag from `docker-compose.yml`
+- rotate `DEPLOY_WEBHOOK_SECRET` when compromise/drift is suspected and sync with GitHub secret
+
+4. Redeploy and validate
+
+- pull latest: `git pull --ff-only origin main`
+- run deploy script with explicit env context
+- verify containers are healthy and smoke endpoints pass:
+    - `/api/health`, `/api/health/auth-config`, `/api/auth/discord`, `/install`, `/legal`, `/discovery`
+
+5. Verify outcome, not just workflow status
+
+- compare running image revisions to intended deploy target
+- if webhook tests are noisy due edge timeouts, re-test directly against webhook container IP (`:9000/hooks/deploy`) to confirm true command outcome
+
+## Incident guardrails
+
+- Treat concurrent deploy agents as a blocker; wait for a quiet window before mutation steps.
+- Preserve evidence (`curl` bodies/status codes, deploy output snippets, container revision checks) in handoff notes.
+- If lock file PID is stale/reused, validate process command line before honoring lock ownership.
diff --git a/.cursor/skills/lucky-docker-dev/SKILL.md b/.cursor/skills/lucky-docker-dev/SKILL.md
@@ -3,29 +3,44 @@ name: lucky-docker-dev
 description: Run and test Lucky with Docker. Use when changing docker-compose, Dockerfiles, or local dev/CI runs.
 ---
 
-# Lucky Docker & Local Dev
+# Lucky Docker + Deploy Validation
 
 ## When to use
 
-- Changing `docker-compose.yml`, `docker-compose.dev.yml`, or Dockerfiles
-- Running bot, backend, or full stack locally
-- Debugging container or env issues
-
-## Layout
-
-- **Compose**: `docker-compose.yml` (prod-like), `docker-compose.dev.yml` (local dev)
-- **Images**: `Dockerfile` (bot/backend), `Dockerfile.frontend`, `Dockerfile.nginx`
-- **Nginx**: `nginx/nginx.conf` — `/api/*` → backend; `/*` → frontend
-
-## Services (typical)
-
-- postgres, redis, bot, backend, frontend, nginx. Env via `.env`; no hardcoded ports or secrets in compose.
-
-## Scripts
-
-- Prefer root `scripts/` (e.g. `setup-database.sh`, `discord-bot.sh`) for one-off or documented flows.
-- Use npm scripts for build/test/lint: `npm run build`, `npm run dev:bot`, `npm run test`, etc.
-
-## Conventions
-
-- Use Docker for local when the project provides it; ensure `.env.example` documents required vars for Docker and local runs.
+- Changing `docker-compose.yml`, `docker-compose.dev.yml`, Dockerfiles, or `scripts/deploy.sh`
+- Deploy verification when webhook/CI looks healthy but runtime is stale
+- Troubleshooting env/secret resolution issues in containerized runs
+
+## Production preflight (required)
+
+1. Resolve required compose vars before deploy:
+    - ensure `.env` has non-empty required values (especially `POSTGRES_PASSWORD`, `DEPLOY_WEBHOOK_SECRET`)
+2. Validate compose rendering before pull/build:
+    - `docker compose --project-directory /home/luk-server/Lucky --env-file /home/luk-server/Lucky/.env config`
+3. If preflight fails, stop and fix env first. Do not continue to pull/deploy.
+
+## Deploy + revision checks
+
+1. Sync repo and deploy:
+    - `git pull --ff-only origin main`
+    - `DEPLOY_WEBHOOK_SECRET=... DEPLOY_DIR=/home/luk-server/Lucky COMPOSE_WORKDIR=/home/luk-server/Lucky ./scripts/deploy.sh "$DEPLOY_WEBHOOK_SECRET"`
+2. Verify runtime revision after rollout:
+    - `git rev-parse --short HEAD`
+    - `docker image inspect -f '{{ index .Config.Labels "org.opencontainers.image.revision" }}' $(docker inspect -f '{{.Image}}' lucky-backend)`
+    - repeat for `lucky-bot`, `lucky-frontend`, `lucky-nginx`
+3. Verify smoke contracts:
+    - `/api/health` -> `200` and `"status":"ok"`
+    - `/api/health/auth-config` -> `200` and auth payload
+    - `/api/auth/discord` -> `302` redirect to Discord OAuth URL
+    - frontend routes `/install`, `/legal`, `/discovery` -> `200`
+
+## Webhook failure-signaling checks
+
+1. `deploy/hooks.json` must keep:
+    - `include-command-output-in-response: true`
+    - `include-command-output-in-response-on-error: true`
+2. Webhook service command should avoid verbose request logging (`-verbose` removed).
+3. Validation:
+    - wrong secret request -> non-2xx with explicit error
+    - correct secret request -> waits for deploy command completion and returns command output
+4. If proxy timeout interferes, test direct webhook endpoint via container IP (`http://<webhook-ip>:9000/hooks/deploy`) to isolate signaling from edge timeouts.
diff --git a/AGENTS.md b/AGENTS.md
@@ -20,22 +20,23 @@ At the start of every session:
 
 ### Project skills (`.cursor/skills/`)
 
-| Task                                              | Skill                    |
-| ------------------------------------------------- | ------------------------ |
-| Add/change slash command                          | `discord-commands`       |
-| Play/queue/skip/volume, player lifecycle          | `music-queue-player`     |
-| Schema, migrations, DB/Redis in shared            | `prisma-redis-lucky`     |
-| Docker, compose, local run                        | `lucky-docker-dev`       |
-| Frontend (React, Vite, Tailwind)                  | `frontend-react-vite`    |
-| Backend (Express API, routes, services)           | `backend-express`        |
-| E2E tests, Playwright, browser verification       | `e2e-playwright`         |
-| Docs lookup, web search, MCP usage                | `mcp-docs-search`        |
-| Moderation commands + AutoModService              | `moderation-automod`     |
-| Bot event wiring (messageCreate, memberAdd, etc.) | `event-handlers`         |
-| Embed builder, custom commands, auto-messages     | `management-features`    |
-| Unit tests, Jest ESM mocks, fixing disabled tests | `testing-lucky`          |
-| CI gate triage and required-check recovery        | `lucky-ci-gate-recovery` |
-| Recover GitHub MCP transport/auth failures        | `mcp-github-recovery`    |
+| Task                                              | Skill                      |
+| ------------------------------------------------- | -------------------------- |
+| Add/change slash command                          | `discord-commands`         |
+| Play/queue/skip/volume, player lifecycle          | `music-queue-player`       |
+| Schema, migrations, DB/Redis in shared            | `prisma-redis-lucky`       |
+| Docker, compose, local run                        | `lucky-docker-dev`         |
+| Workflow green but production stale deploy drift  | `lucky-deploy-recovery`    |
+| Frontend (React, Vite, Tailwind)                  | `frontend-react-vite`      |
+| Backend (Express API, routes, services)           | `backend-express`          |
+| E2E tests, Playwright, browser verification       | `e2e-playwright`           |
+| Docs lookup, web search, MCP usage                | `mcp-docs-search`          |
+| Moderation commands + AutoModService              | `moderation-automod`       |
+| Bot event wiring (messageCreate, memberAdd, etc.) | `event-handlers`           |
+| Embed builder, custom commands, auto-messages     | `management-features`      |
+| Unit tests, Jest ESM mocks, fixing disabled tests | `testing-lucky`            |
+| CI gate triage and required-check recovery        | `lucky-ci-gate-recovery`   |
+| Recover GitHub MCP transport/auth failures        | `mcp-github-recovery`      |
 | OpenCode config, plugins, attach, verification    | `opencode-lucky-workflows` |
 
 ### Ecosystem skills (`.agent-skills/` — from skills.sh)
@@ -144,7 +145,6 @@ Use the **specific specialist and skills** for the task; use **MCP tools** to fi
 
 Use Docker for local when available (`docker-compose.dev.yml`). Prefer scripts in `scripts/` for documented operations.
 
-
 - Repo-local OpenCode behavior lives in `opencode.jsonc`, `.opencode/plugins`, and `.opencode/skills`.
 - Host-local OpenCode auth, provider state, and MCP credentials stay in `~/.config/opencode`.
 - Use `scripts/opencode-sync-project-skills.sh` after changing project skills or OpenCode skill bridges.

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,11 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- Added deploy-recovery skill `.cursor/skills/lucky-deploy-recovery/SKILL.md`
+  and linked it in `AGENTS.md` for "workflow green, production stale" incidents.
+- Expanded project skills:
+  - `.cursor/skills/lucky-docker-dev/SKILL.md` with deploy preflight, revision
+    verification, and webhook signaling validation.
+  - `.cursor/skills/lucky-ci-gate-recovery/SKILL.md` with webhook failure
+    signatures (`dirty-tree-overwrite`, lock-collision, timeout-noise).
+- Added `docs/AUTH_SMOKE_RUNBOOK.md` with manual Discord login smoke workflow
+  and timestamped evidence capture template.
+
 ### Fixed
 
 - Pinned webhook-driven `prisma migrate deploy` and `prisma migrate status`
   calls to `prisma/prisma.config.ts` so homelab deploys keep `DATABASE_URL`
   resolution when run from the webhook container.
+- Deploy webhook contract now includes command output in both success and error
+  responses (`include-command-output-in-response*`) to prevent false-positive
+  trigger results.
+- Removed webhook runtime `-verbose` logging from compose service command to
+  reduce request-secret exposure risk in logs.
+- Hardened `scripts/deploy.sh` with:
+  - required compose-env preflight before deploy actions
+  - host-safe health endpoint resolution for non-container execution context
+  - dirty tracked-worktree fail-fast before `git pull`
+  - robust lock handling that validates stale/reused lock PIDs by command line
 
 ## [2.6.15] - 2026-03-14
 

diff --git a/deploy/hooks.json b/deploy/hooks.json
@@ -4,6 +4,8 @@
         "execute-command": "/home/luk-server/Lucky/scripts/deploy.sh",
         "command-working-directory": "/home/luk-server/Lucky",
         "response-message": "Deploy triggered",
+        "include-command-output-in-response": true,
+        "include-command-output-in-response-on-error": true,
         "pass-arguments-to-command": [
             {
                 "source": "header",

diff --git a/docker-compose.yml b/docker-compose.yml
@@ -173,7 +173,6 @@ services:
         restart: unless-stopped
         command: >
             -hooks /etc/webhook/hooks.json
-            -verbose
             -hotreload
         environment:
             - DEPLOY_WEBHOOK_SECRET=${DEPLOY_WEBHOOK_SECRET}

diff --git a/docs/AUTH_SMOKE_RUNBOOK.md b/docs/AUTH_SMOKE_RUNBOOK.md
@@ -0,0 +1,38 @@
+# Discord Auth Smoke Runbook
+
+Use this runbook after deploy changes that touch webhook/auth/deploy flow.
+
+## Preconditions
+
+- Production deploy already completed successfully.
+- `https://lucky.lucassantana.tech` is reachable.
+- Discord app credentials are configured and `/api/health/auth-config` returns `status=ok`.
+
+## Manual steps (real browser)
+
+1. Open `https://lucky.lucassantana.tech/api/auth/discord`.
+2. Complete Discord login + authorization.
+3. Confirm redirect callback lands in Lucky dashboard.
+4. Confirm dashboard bootstrap works:
+    - guild selector loads
+    - settings/features views render
+    - Twitch notification settings page loads
+5. Verify session status transition:
+    - authenticated after login: `GET /api/auth/status` -> `{"authenticated": true, ...}`
+    - unauthenticated after logout/session clear: `GET /api/auth/status` -> `{"authenticated": false}`
+
+## Evidence log (2026-03-14)
+
+Timestamp (UTC): `2026-03-14T23:32:22Z`
+
+Automated pre-check evidence captured:
+
+- `http://127.0.0.1:8090/api/health` -> `HTTP 200`, body includes `"status":"ok"`
+- `http://127.0.0.1:8090/api/health/auth-config` -> `HTTP 200`, body includes auth contract payload
+- `http://127.0.0.1:8090/api/auth/discord` -> `HTTP 302` to Discord authorize URL
+- `http://127.0.0.1:8090/api/auth/status` -> `{"authenticated":false}` before manual login
+
+Manual browser-login evidence status:
+
+- `PENDING OPERATOR EXECUTION`: real Discord login verification requires interactive operator session and Discord credentials/2FA approval.
+- Agent-side browser automation was unavailable in this environment due missing Playwright Chrome install permissions.