Bump migration timeout to 4h for hypertable backfills

.github/workflows/deploy-web-stack.yml

@@ -412,8 +412,37 @@ jobs:
           docker logs --follow "$migration_container" &
           migration_log_pid=$!
 
-          migration_timeout_seconds=3300
+          # Heavy data migrations (e.g. hypertable backfills moving body
+          # measurements into fitness.metric_stream) can legitimately run for
+          # hours on prod. Sized to fit the slowest known migration with
+          # headroom; if a future migration would exceed this, restructure it
+          # rather than bumping again. The pg_stat_activity snapshots below
+          # produce direct evidence of which statement is running so a long
+          # runtime can be distinguished from lock contention without waiting
+          # for the timeout to fire.
+          migration_timeout_seconds=14400
           migration_start_seconds=$SECONDS
+          next_pg_snapshot_seconds=0
+          snapshot_pg_activity() {
+            local label="$1"
+            echo "--- pg_stat_activity snapshot (${label}) ---"
+            timeout 15s docker run --rm --network "${STACK_NAME}_default" \
+              --env-file "$INFISICAL_ENV_FILE" \
+              postgres:18-alpine sh -euc \
+              "PGPASSWORD=\"\$POSTGRES_PASSWORD\" \
+              psql --host=db --username=health --dbname=health --expanded \
+              --command \"SELECT pid, state, wait_event_type, wait_event, \
+                           now() - xact_start AS xact_age, \
+                           now() - query_start AS query_age, \
+                           left(query, 400) AS query \
+                         FROM pg_stat_activity \
+                         WHERE state IS DISTINCT FROM 'idle' \
+                           AND backend_type = 'client backend' \
+                           AND pid <> pg_backend_pid() \
+                         ORDER BY query_start;\"" \
+              2>&1 || echo "(pg_stat_activity snapshot failed; continuing)"
+            echo "--- end snapshot ---"
+          }
           while true; do
             inspect_error_file="$RUNNER_TEMP/migrate-inspect.log"
             set +e
@@ -440,13 +469,18 @@ jobs:
             elapsed_seconds=$((SECONDS - migration_start_seconds))
             if [ "$elapsed_seconds" -ge "$migration_timeout_seconds" ]; then
               echo "::error::Migration exceeded ${migration_timeout_seconds}s"
+              snapshot_pg_activity "at timeout, ${elapsed_seconds}s"
               echo "Migration output before timeout (last 120 lines):"
               timeout 60s docker logs --tail 120 "$migration_container" || true
               cleanup_migration_container
               exit 1
             fi
 
             echo "Migration still running after ${elapsed_seconds}s..."
+            if [ "$elapsed_seconds" -ge "$next_pg_snapshot_seconds" ]; then
+              snapshot_pg_activity "${elapsed_seconds}s elapsed"
+              next_pg_snapshot_seconds=$((elapsed_seconds + 60))
+            fi
             sleep 15
           done
 

docs/production-incident-baseline.md

@@ -7,6 +7,167 @@ full incident log or a replacement for runbooks. Use it to build shared memory
 about the kinds of issues this system encounters, the signals that identified
 them, and the durability work they suggest.
 
+## 2026-05-10: Prod host wedged after rescue-mode reboot during deploy unblock attempt
+
+### Impact
+
+`dofek.asherlc.com` went fully unreachable from the internet. Both prod
+(`dofek` / 157.90.25.125) and `dofek-staging` (162.55.186.24) now exhibit
+the same wedged-host pattern: TCP accepts on `:22` and `:443`, but neither
+sshd nor Traefik return a response. Multiple soft reboots and a hard
+`hcloud server reset` did not produce a stable host — each boot yields ~1–3
+minutes of SSH responsiveness, then the host becomes unresponsive again.
+
+The originally-intended action was a one-shot `docker service update --image
+timescale/timescaledb-ha:pg18.3-ts2.26.4-all dofek_db` to give the prod DB
+container the postgis extension that the new
+`drizzle/0018_metric_stream_location_point.sql` migration (from PR #1111)
+requires. Whether that swap completed before the host wedged is unknown —
+the SSH session ran the command but never returned output.
+
+### Evidence That Mattered
+
+- During the brief windows of SSH responsiveness post-reboot:
+  - `uptime` reported 1-min load average climbing from `5.47` (right after
+    boot) to `111.23` on a 2-core (`cax11`) ARM instance within ~9 minutes.
+  - `docker stack services dofek` listed ~20 services starting concurrently:
+    TimescaleDB, ClickHouse, Redis, Traefik, web, training-export-worker, ml,
+    otel-collector, netdata, portainer, cloudbeaver, databasus, pgadmin, the
+    PeerDB cluster (`peerdb`, `flow-api`, `flow-snapshot-worker`,
+    `flow-worker`, `temporal`, `temporal-admin-tools`, `minio`, `catalog`),
+    ota, docuum.
+  - Several services (`db`, `redis`, all `peerdb-*-worker`, `peerdb-temporal`,
+    `pgadmin`, `netdata`, `docuum`) were stuck at `0/1` replicas, never
+    reaching healthy.
+- hcloud metrics during wedged windows: CPU baseline (no saturation pattern
+  visible in the chart), disk IOPS in `1–5` range, network sub-1 MB/s. Low
+  metrics combined with high load average indicates many processes blocked
+  in `D` state (uninterruptible IO wait) on the network-attached `dofek-data`
+  volume — not active CPU/IO consumption.
+- Identical symptom on `dofek-staging` from the start of the session,
+  predating any of today's actions.
+
+### Root Cause
+
+Provisional, not fully confirmed because the host stays unreachable long
+enough only for shallow inspection: the dofek swarm stack appears to be
+oversubscribed for the `cax11` instance class. Boot triggers a thundering
+herd of ~20 services starting concurrently, all contending for the
+network-attached data volume. Load average climbs to >100 on a 2-core box,
+sshd cannot get scheduled for new pre-auth forks, Traefik likewise stops
+serving, and the host remains in this state until something gives. Soft
+reboots simply restart the same cascade.
+
+The PeerDB cluster (8 services including a full Temporal control plane) is
+the most likely structural contributor — it adds substantial steady-state
+RAM and IO load that is independent of the actual app workload.
+
+The deploy-unblock action (`docker service update` to swap the db image)
+was not the cause — the host already exhibits this pattern on staging
+without any deploy intervention. But the rescue → reboot dance did move
+prod from "responsive" into the same wedged state staging was already in.
+
+### Fix Or Mitigation
+
+Unresolved at time of writing. Realistic paths:
+
+1. **Resize the host**: `hcloud server change-type dofek cax21` (4 vCPU /
+   8 GB) or `cax31` (8 vCPU / 16 GB). Requires a stop+start. Apply to
+   `dofek-staging` likewise.
+2. **Trim the stack**: scale non-essential services to 0 while the host
+   recovers, e.g. `docker service scale dofek_peerdb-flow-worker=0
+   dofek_peerdb-flow-snapshot-worker=0 dofek_peerdb-temporal=0
+   dofek_peerdb-temporal-admin-tools=0 dofek_peerdb-flow-api=0
+   dofek_docuum=0 dofek_pgadmin=0 dofek_netdata=0`. Bring them back
+   incrementally after the host stabilizes. Requires SSH access during a
+   responsive window.
+3. **Both**, in series: resize first (gets a steady-state box), then
+   investigate whether all PeerDB / observability services are still needed.
+
+The previously-merged `migration_timeout_seconds=14400` bump and
+`pg_stat_activity` instrumentation in PR #1113 are unaffected and remain
+valid groundwork — but cannot be exercised until the host is stable.
+
+### Remaining Risk
+
+- We do not yet know whether the post-reboot wedge is a deterministic
+  consequence of the stack composition, or whether something specific to
+  the data volume (slow IO, near-full, corruption, etc.) is the proximate
+  cause. Resizing the host treats the symptom; if the underlying issue is
+  the volume, resize alone will not durably help.
+- Staging has been wedged through the entire incident, so the deploy-time
+  validation surface is also down. Any fix attempted on prod cannot be
+  rehearsed on staging until staging is recovered.
+- The pending `0018_metric_stream_location_point.sql` and
+  `0018_migrate_body_measurements_to_metric_stream.sql` migrations are
+  still blocked from applying. The deploy queue is effectively frozen.
+
+## 2026-05-09: Production deploy timed out applying body-measurement hypertable migration
+
+### Impact
+
+Deploy Web run `25609852303` from `main` (commit `f17e27d`) failed: the prod
+job's `Run migrations` step was killed at `3300s` while applying
+`drizzle/0018_migrate_body_measurements_to_metric_stream.sql`. The migration
+ran inside a transaction with no statement breakpoints, so the force-killed
+container rolled back; prod DB stayed in the pre-migration state and code did
+not roll. Staging deploy in the same workflow run failed independently with an
+SSH banner timeout to `162.55.186.24` (separate incident, handled out of
+band).
+
+### Evidence That Mattered
+
+- Job: `Deploy Web Production / Deploy Web Stack / Deploy Web Stack`.
+- Step: `Run migrations`, last log lines:
+  `info: [migrate] Applying: 0018_migrate_body_measurements_to_metric_stream.sql`
+  → repeated `Migration still running after Ns...` (no progress within-step
+  visibility) → `##[error]Migration exceeded 3300s` at `2026-05-09T20:36:59Z`.
+- The migration is a single transactional block over `fitness.metric_stream`
+  (TimescaleDB hypertable): UPDATE backfilling `external_id`, DELETE of
+  duplicate channel rows, `CREATE UNIQUE INDEX`, and `INSERT … SELECT` from
+  `fitness.body_measurement` covering 12 body channels. No
+  `--no-transaction` / `statement-breakpoint` markers, so all-or-nothing.
+- Prod host (`157.90.25.125`) was healthy throughout: SSH banner returned in
+  `~2s`, `dofek.asherlc.com/healthz` returned `200` directly against the
+  origin during and after the failure. The migration was making progress and
+  hit the deploy-level guard, not a DB error.
+
+### Root Cause
+
+`migration_timeout_seconds=3300` in `.github/workflows/deploy-web-stack.yml`
+was sized for routine schema migrations and is too small for a one-shot
+hypertable backfill that copies all `fitness.body_measurement` rows into
+`fitness.metric_stream` and rebuilds dependent indexes/views in the same
+transaction.
+
+### Fix Or Mitigation
+
+- Bumped `migration_timeout_seconds` from `3300` (55min) to `14400` (4h) in
+  `.github/workflows/deploy-web-stack.yml` so the migration has room to
+  complete on the next deploy attempt.
+- Added a comment at the constant explaining that future migrations exceeding
+  this budget should be restructured (split, batched with
+  `--no-transaction`, or moved to an out-of-band backfill job) rather than
+  bumping the timeout again.
+- Added periodic `pg_stat_activity` snapshots (every 60s of elapsed
+  migration time, plus a final snapshot on timeout) to the workflow log so
+  the next failure produces direct evidence of which SQL statement is
+  running and whether it is progressing or stuck on a `wait_event` (lock,
+  IO, etc.) — without waiting for the timeout to fire.
+
+### Remaining Risk
+
+We do not have direct evidence of how long `0018` actually needs to run to
+completion on prod — only that it was still progressing at 55min. If 4h is
+also insufficient, the migration is likely unbounded and must be
+restructured: split into `ADD COLUMN` (fast) + chunked, non-transactional
+UPDATE/DELETE/INSERT batches keyed on TimescaleDB chunk boundaries +
+post-backfill DDL (`CREATE UNIQUE INDEX`, view rebuilds, `DROP TABLE`).
+Splitting cleanly also requires app code in
+`packages/server/src/repositories/body-*.ts` to read from both
+`fitness.body_measurement` and the `metric_stream` body channels during the
+backfill window, which is a multi-deploy refactor.
+
 ## 2026-05-06: Branch deploy validation exceeded migration timeout
 
 ### Impact