From eb70c682c73e14f2993ab91a98aa730ade26f922 Mon Sep 17 00:00:00 2001 From: LucasSantana-Dev Date: Sat, 14 Mar 2026 20:29:42 -0300 Subject: [PATCH 1/2] fix(deploy): harden webhook preflight and recovery skills --- .../skills/lucky-ci-gate-recovery/SKILL.md | 16 ++ .cursor/skills/lucky-deploy-recovery/SKILL.md | 52 ++++++ .cursor/skills/lucky-docker-dev/SKILL.md | 61 ++++--- AGENTS.md | 34 ++-- deploy/hooks.json | 2 + docker-compose.yml | 1 - scripts/deploy.sh | 162 ++++++++++++++++-- 7 files changed, 274 insertions(+), 54 deletions(-) create mode 100644 .cursor/skills/lucky-deploy-recovery/SKILL.md diff --git a/.cursor/skills/lucky-ci-gate-recovery/SKILL.md b/.cursor/skills/lucky-ci-gate-recovery/SKILL.md index ab48d2ced..88e87c79e 100644 --- a/.cursor/skills/lucky-ci-gate-recovery/SKILL.md +++ b/.cursor/skills/lucky-ci-gate-recovery/SKILL.md @@ -63,6 +63,22 @@ gh pr merge --squash --match-head-commit "$SHA" - Dependabot runs without token: scanner path must skip as success - Required status names must remain stable with ruleset contexts +## Deploy webhook failure signatures + +When CI/CD push checks are green but `Deploy to Homelab` fails in webhook trigger, classify quickly: + +- `dirty-tree-overwrite`: + - body includes `error: Your local changes to the following files would be overwritten by merge` + - fix: clean tracked workspace changes on host, then retry deploy +- `deploy-lock-collision`: + - body includes `ERROR: another deploy is already running` + - fix: wait for active deploy to finish; if stale lock suspected, verify lock PID command line before clearing +- `edge-timeout-noise`: + - public webhook path returns timeout/504 while deploy may continue + - fix: validate same request directly against webhook container endpoint (`http://:9000/hooks/deploy`) and confirm command output there + +Use these signatures before changing workflow retry logic. + ## Deterministic status policy - Only statuses listed in required checks gate merge decisions. diff --git a/.cursor/skills/lucky-deploy-recovery/SKILL.md b/.cursor/skills/lucky-deploy-recovery/SKILL.md new file mode 100644 index 000000000..16d8b7e55 --- /dev/null +++ b/.cursor/skills/lucky-deploy-recovery/SKILL.md @@ -0,0 +1,52 @@ +--- +name: lucky-deploy-recovery +description: Recover Lucky when workflow checks pass but production stays stale, webhook deploys fail, or deploy signaling is inconsistent. +--- + +# Lucky Deploy Recovery + +## When to use + +- GitHub deploy workflow is green but production revision is old +- Webhook returns success while deploy command failed (or never started) +- Deploys fail on missing compose secrets, lock collisions, or stale runtime drift + +## Recovery workflow + +1. Capture baseline + +- `git rev-parse --short HEAD` +- `git rev-parse --short origin/main` +- container revisions from `org.opencontainers.image.revision` for backend/bot/frontend/nginx + +2. Repair deploy prerequisites + +- restore/confirm required `.env` values, especially `POSTGRES_PASSWORD` and `DEPLOY_WEBHOOK_SECRET` +- run compose preflight before deploy: + - `docker compose --project-directory /home/luk-server/Lucky --env-file /home/luk-server/Lucky/.env config` + +3. Harden webhook signaling + +- in `deploy/hooks.json`, keep: + - `include-command-output-in-response: true` + - `include-command-output-in-response-on-error: true` +- remove webhook `-verbose` runtime flag from `docker-compose.yml` +- rotate `DEPLOY_WEBHOOK_SECRET` when compromise/drift is suspected and sync with GitHub secret + +4. Redeploy and validate + +- pull latest: `git pull --ff-only origin main` +- run deploy script with explicit env context +- verify containers are healthy and smoke endpoints pass: + - `/api/health`, `/api/health/auth-config`, `/api/auth/discord`, `/install`, `/legal`, `/discovery` + +5. Verify outcome, not just workflow status + +- compare running image revisions to intended deploy target +- if webhook tests are noisy due edge timeouts, re-test directly against webhook container IP (`:9000/hooks/deploy`) to confirm true command outcome + +## Incident guardrails + +- Treat concurrent deploy agents as a blocker; wait for a quiet window before mutation steps. +- Preserve evidence (`curl` bodies/status codes, deploy output snippets, container revision checks) in handoff notes. +- If lock file PID is stale/reused, validate process command line before honoring lock ownership. diff --git a/.cursor/skills/lucky-docker-dev/SKILL.md b/.cursor/skills/lucky-docker-dev/SKILL.md index 732abf928..a9a645000 100644 --- a/.cursor/skills/lucky-docker-dev/SKILL.md +++ b/.cursor/skills/lucky-docker-dev/SKILL.md @@ -3,29 +3,44 @@ name: lucky-docker-dev description: Run and test Lucky with Docker. Use when changing docker-compose, Dockerfiles, or local dev/CI runs. --- -# Lucky Docker & Local Dev +# Lucky Docker + Deploy Validation ## When to use -- Changing `docker-compose.yml`, `docker-compose.dev.yml`, or Dockerfiles -- Running bot, backend, or full stack locally -- Debugging container or env issues - -## Layout - -- **Compose**: `docker-compose.yml` (prod-like), `docker-compose.dev.yml` (local dev) -- **Images**: `Dockerfile` (bot/backend), `Dockerfile.frontend`, `Dockerfile.nginx` -- **Nginx**: `nginx/nginx.conf` — `/api/*` → backend; `/*` → frontend - -## Services (typical) - -- postgres, redis, bot, backend, frontend, nginx. Env via `.env`; no hardcoded ports or secrets in compose. - -## Scripts - -- Prefer root `scripts/` (e.g. `setup-database.sh`, `discord-bot.sh`) for one-off or documented flows. -- Use npm scripts for build/test/lint: `npm run build`, `npm run dev:bot`, `npm run test`, etc. - -## Conventions - -- Use Docker for local when the project provides it; ensure `.env.example` documents required vars for Docker and local runs. +- Changing `docker-compose.yml`, `docker-compose.dev.yml`, Dockerfiles, or `scripts/deploy.sh` +- Deploy verification when webhook/CI looks healthy but runtime is stale +- Troubleshooting env/secret resolution issues in containerized runs + +## Production preflight (required) + +1. Resolve required compose vars before deploy: + - ensure `.env` has non-empty required values (especially `POSTGRES_PASSWORD`, `DEPLOY_WEBHOOK_SECRET`) +2. Validate compose rendering before pull/build: + - `docker compose --project-directory /home/luk-server/Lucky --env-file /home/luk-server/Lucky/.env config` +3. If preflight fails, stop and fix env first. Do not continue to pull/deploy. + +## Deploy + revision checks + +1. Sync repo and deploy: + - `git pull --ff-only origin main` + - `DEPLOY_WEBHOOK_SECRET=... DEPLOY_DIR=/home/luk-server/Lucky COMPOSE_WORKDIR=/home/luk-server/Lucky ./scripts/deploy.sh "$DEPLOY_WEBHOOK_SECRET"` +2. Verify runtime revision after rollout: + - `git rev-parse --short HEAD` + - `docker image inspect -f '{{ index .Config.Labels "org.opencontainers.image.revision" }}' $(docker inspect -f '{{.Image}}' lucky-backend)` + - repeat for `lucky-bot`, `lucky-frontend`, `lucky-nginx` +3. Verify smoke contracts: + - `/api/health` -> `200` and `"status":"ok"` + - `/api/health/auth-config` -> `200` and auth payload + - `/api/auth/discord` -> `302` redirect to Discord OAuth URL + - frontend routes `/install`, `/legal`, `/discovery` -> `200` + +## Webhook failure-signaling checks + +1. `deploy/hooks.json` must keep: + - `include-command-output-in-response: true` + - `include-command-output-in-response-on-error: true` +2. Webhook service command should avoid verbose request logging (`-verbose` removed). +3. Validation: + - wrong secret request -> non-2xx with explicit error + - correct secret request -> waits for deploy command completion and returns command output +4. If proxy timeout interferes, test direct webhook endpoint via container IP (`http://:9000/hooks/deploy`) to isolate signaling from edge timeouts. diff --git a/AGENTS.md b/AGENTS.md index 6b675b681..532c8cb69 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,22 +20,23 @@ At the start of every session: ### Project skills (`.cursor/skills/`) -| Task | Skill | -| ------------------------------------------------- | ------------------------ | -| Add/change slash command | `discord-commands` | -| Play/queue/skip/volume, player lifecycle | `music-queue-player` | -| Schema, migrations, DB/Redis in shared | `prisma-redis-lucky` | -| Docker, compose, local run | `lucky-docker-dev` | -| Frontend (React, Vite, Tailwind) | `frontend-react-vite` | -| Backend (Express API, routes, services) | `backend-express` | -| E2E tests, Playwright, browser verification | `e2e-playwright` | -| Docs lookup, web search, MCP usage | `mcp-docs-search` | -| Moderation commands + AutoModService | `moderation-automod` | -| Bot event wiring (messageCreate, memberAdd, etc.) | `event-handlers` | -| Embed builder, custom commands, auto-messages | `management-features` | -| Unit tests, Jest ESM mocks, fixing disabled tests | `testing-lucky` | -| CI gate triage and required-check recovery | `lucky-ci-gate-recovery` | -| Recover GitHub MCP transport/auth failures | `mcp-github-recovery` | +| Task | Skill | +| ------------------------------------------------- | -------------------------- | +| Add/change slash command | `discord-commands` | +| Play/queue/skip/volume, player lifecycle | `music-queue-player` | +| Schema, migrations, DB/Redis in shared | `prisma-redis-lucky` | +| Docker, compose, local run | `lucky-docker-dev` | +| Workflow green but production stale deploy drift | `lucky-deploy-recovery` | +| Frontend (React, Vite, Tailwind) | `frontend-react-vite` | +| Backend (Express API, routes, services) | `backend-express` | +| E2E tests, Playwright, browser verification | `e2e-playwright` | +| Docs lookup, web search, MCP usage | `mcp-docs-search` | +| Moderation commands + AutoModService | `moderation-automod` | +| Bot event wiring (messageCreate, memberAdd, etc.) | `event-handlers` | +| Embed builder, custom commands, auto-messages | `management-features` | +| Unit tests, Jest ESM mocks, fixing disabled tests | `testing-lucky` | +| CI gate triage and required-check recovery | `lucky-ci-gate-recovery` | +| Recover GitHub MCP transport/auth failures | `mcp-github-recovery` | | OpenCode config, plugins, attach, verification | `opencode-lucky-workflows` | ### Ecosystem skills (`.agent-skills/` — from skills.sh) @@ -144,7 +145,6 @@ Use the **specific specialist and skills** for the task; use **MCP tools** to fi Use Docker for local when available (`docker-compose.dev.yml`). Prefer scripts in `scripts/` for documented operations. - - Repo-local OpenCode behavior lives in `opencode.jsonc`, `.opencode/plugins`, and `.opencode/skills`. - Host-local OpenCode auth, provider state, and MCP credentials stay in `~/.config/opencode`. - Use `scripts/opencode-sync-project-skills.sh` after changing project skills or OpenCode skill bridges. diff --git a/deploy/hooks.json b/deploy/hooks.json index 60061f46a..39f804827 100644 --- a/deploy/hooks.json +++ b/deploy/hooks.json @@ -4,6 +4,8 @@ "execute-command": "/home/luk-server/Lucky/scripts/deploy.sh", "command-working-directory": "/home/luk-server/Lucky", "response-message": "Deploy triggered", + "include-command-output-in-response": true, + "include-command-output-in-response-on-error": true, "pass-arguments-to-command": [ { "source": "header", diff --git a/docker-compose.yml b/docker-compose.yml index 192004d3a..ae6d2a68b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -173,7 +173,6 @@ services: restart: unless-stopped command: > -hooks /etc/webhook/hooks.json - -verbose -hotreload environment: - DEPLOY_WEBHOOK_SECRET=${DEPLOY_WEBHOOK_SECRET} diff --git a/scripts/deploy.sh b/scripts/deploy.sh index 7e56c23a9..fc7a8c7ae 100755 --- a/scripts/deploy.sh +++ b/scripts/deploy.sh @@ -15,6 +15,20 @@ export COMPOSE_PROJECT_NAME log() { echo "$LOG_PREFIX $(date '+%H:%M:%S') $1"; } +env_file_value() { + local key="$1" + local env_file="$2" + + [[ -f "$env_file" ]] || return 1 + + awk -F= -v target="$key" ' + $1 == target { + print substr($0, index($0, "=") + 1) + exit + } + ' "$env_file" +} + resolve_cloudflared_config_dir() { if [[ -n "${CLOUDFLARED_CONFIG_DIR:-}" ]]; then echo "$CLOUDFLARED_CONFIG_DIR" @@ -169,19 +183,101 @@ resolve_postgres_password() { fi local env_file="$COMPOSE_WORKDIR/.env" - if [[ -f "$env_file" ]]; then - local value - value=$(awk -F= '/^POSTGRES_PASSWORD=/{print substr($0, index($0, "=") + 1); exit}' \ - "$env_file") - if [[ -n "$value" ]]; then - echo "$value" - return - fi + local value + value="$(env_file_value "POSTGRES_PASSWORD" "$env_file" || true)" + if [[ -n "$value" ]]; then + echo "$value" + return fi echo "" } +collect_required_compose_vars() { + local compose_file="$COMPOSE_WORKDIR/docker-compose.yml" + [[ -f "$compose_file" ]] || return 0 + + grep -oE '\$\{[A-Z0-9_]+\?[^}]*\}' "$compose_file" \ + | sed -E 's/^\$\{([A-Z0-9_]+)\?.*/\1/' \ + | sort -u +} + +require_compose_env_vars() { + local env_file="$COMPOSE_WORKDIR/.env" + local missing=() + local key value + local required=() + + mapfile -t required < <(collect_required_compose_vars) + + for key in "${required[@]}"; do + value="${!key:-}" + if [[ -z "$value" ]]; then + value="$(env_file_value "$key" "$env_file" || true)" + fi + + if [[ -z "$value" ]]; then + missing+=("$key") + continue + fi + + export "$key=$value" + done + + if [[ "${#missing[@]}" -gt 0 ]]; then + log "ERROR: required compose env vars missing: ${missing[*]}" + notify 16711680 "Deploy Failed" "Missing required compose env vars: ${missing[*]}" + return 1 + fi + + return 0 +} + +run_compose_preflight() { + log "Running compose preflight..." + if ! docker_compose config >/dev/null; then + log "ERROR: docker compose preflight failed (invalid config or unresolved vars)" + notify 16711680 "Deploy Failed" "Compose preflight failed" + return 1 + fi + log "Compose preflight passed" + return 0 +} + +resolve_healthcheck_base_url() { + if [[ -n "${HEALTHCHECK_BASE_URL:-}" ]]; then + echo "${HEALTHCHECK_BASE_URL%/}" + return + fi + + if getent hosts nginx >/dev/null 2>&1; then + echo "http://nginx" + return + fi + + local mapped_port + mapped_port="$(docker port lucky-nginx 80/tcp 2>/dev/null | head -n1 | awk -F: '{print $NF}')" + if [[ -n "$mapped_port" ]]; then + echo "http://127.0.0.1:$mapped_port" + return + fi + + if [[ -n "${WEBAPP_BACKEND_URL:-}" ]]; then + echo "${WEBAPP_BACKEND_URL%/}" + return + fi + + local env_file="$COMPOSE_WORKDIR/.env" + local configured_port + configured_port="$(env_file_value "NGINX_PORT" "$env_file" || true)" + if [[ -n "$configured_port" ]]; then + echo "http://127.0.0.1:$configured_port" + return + fi + + echo "http://127.0.0.1:8080" +} + wait_for_http_ready() { local label="$1" local url="$2" @@ -224,7 +320,11 @@ acquire_lock() { fi if [[ -n "$existing_pid" ]] && kill -0 "$existing_pid" 2>/dev/null; then - return 1 + local existing_cmd + existing_cmd=$(ps -p "$existing_pid" -o args= 2>/dev/null || true) + if echo "$existing_cmd" | grep -Eq '(^|[[:space:]])(deploy\.sh|/scripts/deploy\.sh)([[:space:]]|$)'; then + return 1 + fi fi rm -rf "$LOCK_DIR" 2>/dev/null || true @@ -233,6 +333,24 @@ acquire_lock() { return 0 } +fail_if_git_dirty() { + local dirty + dirty=$(git -C "$DEPLOY_DIR" status --porcelain --untracked-files=no || true) + + if [[ -z "$dirty" ]]; then + return 0 + fi + + log "ERROR: working tree has tracked local changes; refusing deploy pull" + while IFS= read -r entry; do + [[ -z "$entry" ]] && continue + log "DIRTY: $entry" + done <<<"$dirty" + log "ERROR: remediate by committing/stashing/discarding local changes, then rerun deploy" + notify 16711680 "Deploy Failed" "Dirty working tree blocks deploy pull" + return 1 +} + if [[ -z "$EXPECTED_SECRET" ]]; then log "ERROR: DEPLOY_WEBHOOK_SECRET not configured" exit 1 @@ -252,7 +370,19 @@ trap 'rm -rf "$LOCK_DIR" 2>/dev/null || true' EXIT COMPOSE_WORKDIR="$(resolve_compose_workdir)" CLOUDFLARED_CONFIG_DIR="$(resolve_cloudflared_config_dir)" -POSTGRES_PASSWORD_EFFECTIVE="$(resolve_postgres_password)" + +if ! require_compose_env_vars; then + exit 1 +fi + +if ! run_compose_preflight; then + exit 1 +fi + +POSTGRES_PASSWORD_EFFECTIVE="${POSTGRES_PASSWORD:-}" +if [[ -z "$POSTGRES_PASSWORD_EFFECTIVE" ]]; then + POSTGRES_PASSWORD_EFFECTIVE="$(resolve_postgres_password)" +fi if [[ -z "$POSTGRES_PASSWORD_EFFECTIVE" ]]; then log "ERROR: POSTGRES_PASSWORD is required" @@ -271,7 +401,10 @@ git config --global --add safe.directory "$DEPLOY_DIR" notify 16776960 "Deploy Started" "Pulling latest changes and rebuilding..." log "Pulling latest changes..." -git pull origin main +if ! fail_if_git_dirty; then + exit 1 +fi +git pull --ff-only origin main log "Pulling images..." if ! docker_compose pull bot backend frontend nginx; then @@ -349,6 +482,9 @@ sleep 10 log "Service status:" docker_compose ps --format "table {{.Name}}\t{{.Status}}" +HEALTHCHECK_BASE_URL="$(resolve_healthcheck_base_url)" +log "Using healthcheck base URL: $HEALTHCHECK_BASE_URL" + if ! require_running_containers; then print_targeted_logs notify 16711680 "Deploy Failed" "Required services are missing or not running" @@ -365,7 +501,7 @@ fi if ! wait_for_http_ready \ "API health" \ - "http://nginx/api/health" \ + "$HEALTHCHECK_BASE_URL/api/health" \ '"status"[[:space:]]*:[[:space:]]*"ok"'; then print_targeted_logs notify 16711680 "Deploy Failed" "API health endpoint did not become ready" @@ -374,7 +510,7 @@ fi if ! wait_for_http_ready \ "Auth config health" \ - "http://nginx/api/health/auth-config" \ + "$HEALTHCHECK_BASE_URL/api/health/auth-config" \ '"auth"[[:space:]]*:'; then print_targeted_logs notify 16711680 "Deploy Failed" "Auth config health endpoint did not become ready" From 20c754f29e55805c7219883c029e9cd14fd86a94 Mon Sep 17 00:00:00 2001 From: LucasSantana-Dev Date: Sat, 14 Mar 2026 20:32:55 -0300 Subject: [PATCH 2/2] docs(auth): add discord smoke runbook and changelog notes --- CHANGELOG.md | 22 ++++++++++++++++++++++ docs/AUTH_SMOKE_RUNBOOK.md | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+) create mode 100644 docs/AUTH_SMOKE_RUNBOOK.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 8978a628e..7d515510f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,11 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- Added deploy-recovery skill `.cursor/skills/lucky-deploy-recovery/SKILL.md` + and linked it in `AGENTS.md` for "workflow green, production stale" incidents. +- Expanded project skills: + - `.cursor/skills/lucky-docker-dev/SKILL.md` with deploy preflight, revision + verification, and webhook signaling validation. + - `.cursor/skills/lucky-ci-gate-recovery/SKILL.md` with webhook failure + signatures (`dirty-tree-overwrite`, lock-collision, timeout-noise). +- Added `docs/AUTH_SMOKE_RUNBOOK.md` with manual Discord login smoke workflow + and timestamped evidence capture template. + ### Fixed - Pinned webhook-driven `prisma migrate deploy` and `prisma migrate status` calls to `prisma/prisma.config.ts` so homelab deploys keep `DATABASE_URL` resolution when run from the webhook container. +- Deploy webhook contract now includes command output in both success and error + responses (`include-command-output-in-response*`) to prevent false-positive + trigger results. +- Removed webhook runtime `-verbose` logging from compose service command to + reduce request-secret exposure risk in logs. +- Hardened `scripts/deploy.sh` with: + - required compose-env preflight before deploy actions + - host-safe health endpoint resolution for non-container execution context + - dirty tracked-worktree fail-fast before `git pull` + - robust lock handling that validates stale/reused lock PIDs by command line ## [2.6.15] - 2026-03-14 diff --git a/docs/AUTH_SMOKE_RUNBOOK.md b/docs/AUTH_SMOKE_RUNBOOK.md new file mode 100644 index 000000000..44097b13e --- /dev/null +++ b/docs/AUTH_SMOKE_RUNBOOK.md @@ -0,0 +1,38 @@ +# Discord Auth Smoke Runbook + +Use this runbook after deploy changes that touch webhook/auth/deploy flow. + +## Preconditions + +- Production deploy already completed successfully. +- `https://lucky.lucassantana.tech` is reachable. +- Discord app credentials are configured and `/api/health/auth-config` returns `status=ok`. + +## Manual steps (real browser) + +1. Open `https://lucky.lucassantana.tech/api/auth/discord`. +2. Complete Discord login + authorization. +3. Confirm redirect callback lands in Lucky dashboard. +4. Confirm dashboard bootstrap works: + - guild selector loads + - settings/features views render + - Twitch notification settings page loads +5. Verify session status transition: + - authenticated after login: `GET /api/auth/status` -> `{"authenticated": true, ...}` + - unauthenticated after logout/session clear: `GET /api/auth/status` -> `{"authenticated": false}` + +## Evidence log (2026-03-14) + +Timestamp (UTC): `2026-03-14T23:32:22Z` + +Automated pre-check evidence captured: + +- `http://127.0.0.1:8090/api/health` -> `HTTP 200`, body includes `"status":"ok"` +- `http://127.0.0.1:8090/api/health/auth-config` -> `HTTP 200`, body includes auth contract payload +- `http://127.0.0.1:8090/api/auth/discord` -> `HTTP 302` to Discord authorize URL +- `http://127.0.0.1:8090/api/auth/status` -> `{"authenticated":false}` before manual login + +Manual browser-login evidence status: + +- `PENDING OPERATOR EXECUTION`: real Discord login verification requires interactive operator session and Discord credentials/2FA approval. +- Agent-side browser automation was unavailable in this environment due missing Playwright Chrome install permissions.