Skip to content

fix(pgserve): canonical cutover — consumer-only model, no more pkill of pm2 - #1635

Merged
namastex888 merged 4 commits into
devfrom
fix/pgserve-canonical-cutover
May 4, 2026
Merged

namastex888 merged 4 commits into
devfrom
fix/pgserve-canonical-cutover

Conversation

@automagik-genie

Copy link
Copy Markdown
Contributor

Summary

  • Converts genie from a pgserve owner to a consumer of the canonical pm2-supervised pgserve daemon. Genie no longer spawns pgserve, never pkill -9s postgres backends, and surfaces canonical-pgserve unavailability with a copy-paste pm2 recovery hint.
  • Closes the bug behind every "Could not kill stale postgres processes" + "pgserve v2 daemon exited before binding" cycle: the pre-cutover selfHealPostgres was fighting pm2's restart-on-crash; every kill triggered an immediate respawn.
  • Net change: −745 LOC in src/lib/db.ts (≈3× surface-area collapse), +220 LOC across new tests, hint helpers, and docs.

Reference: .genie/wishes/pgserve-canonical-cutover/WISH.md.

What changed

G1 — genie install is now fatal on canonical pgserve failure (eefc9a94)

  • tryPgserveInstall → requirePgserveInstall: void on success, exits 1 with a copy-paste hint on any failure (binary missing, install non-zero, port discovery failure).
  • Hint includes the three recovery commands (bun add -g pgserve@^2 / pgserve install / genie install) and links docs/install.md.

G2 — getOrStartDaemon → requirePgserveDaemon (probe-only Mode A) (bd8845eb)

  • Probe via probePgserveDaemon + isPgserveSocketResponsive. On unreachable: throws buildPgserveUnavailableHint (pm2 status / pm2 restart pgserve / pgserve install).
  • The pre-cutover symbol is removed (the project's dead-code gate doesn't honour @deprecated; CHANGELOG documents the rename).

G3 — Delete spawn helpers + replace serve.ts startPgserve (bd8845eb)

  • Deleted from src/lib/db.ts: startPgserveDaemonOnce, evictOrphanDataDirHolder, detectOrphanDataDirLock, terminatePgserveTree, signalPgserveTree, signalPgserveDaemonPid, recoverUnresponsivePgserveDaemon, isLikelyPgserveDaemonProcess, cleanPartialDaemonState, removeStalePgserveSocketArtifacts, unlinkIfPresent, waitForDaemonSocket, formatPgserveDaemonCommand, spawnPgserveDirect, startPgserveOnPort, findPgserveBin, findPgserveDaemonCommand, findLocalPgserveRoot, resolvePgservePackageCommand, findBunRuntime, waitForDaemonPort, throwDaemonTimeout, pgserveChild, maskCredentials, isPgAutostartDisabled, the PgserveDaemonCommand interface, the local sleep, and the lastAutoStartOutcome/lastAutoStartPid tracking.
  • src/term-commands/serve.ts: startPgserve → requirePgserveReady (probe-only). On success: pgserve daemon ready (canonical, pm2-supervised) on <socket>. On failure: pm2-recovery hint + sets GENIE_PG_NO_AUTOSTART=1.
  • _ensurePgserve simplified to fail with the canonical hint when no existing TCP port is reachable. registerExitHandler is now invoked from _buildConnection post-connect so the postgres pool drain still runs on every clean exit.

G4 — Delete selfHealPostgres + remove doctor pkill (bd8845eb)

  • selfHealPostgres deleted (its only callers were the deleted spawn helpers).
  • src/genie-commands/doctor.ts: killStalePostgres → printPgserveRecoveryHint (hint-only; never shells out). doctor --fix no longer prints "Killing stale postgres processes" or "Could not kill".

G5 — Regression coverage (0ae69eb3 + 3f54ed87)

  • New test requirePgserveDaemon never spawns when daemon is healthy (cutover G5 regression) — source-text invariant: function body never references any child_process spawn primitive nor process.kill.
  • New "canonical-cutover removed every spawn helper from db.ts" lockout test asserts every removed symbol is absent.

G6 — Docs

  • README.md: new "Manual install (canonical pgserve first)" subsection with the three-step canonical pattern.
  • CHANGELOG.md: Unreleased entry with breaking-change rationale + a copy-paste migration block (including pgserve install --data ~/.genie/data/pgserve for keeping pre-canonical data dirs).

Side fix uncovered by Wave 3 validation

Moving registerExitHandler() from the (deleted) spawn-side call site into _buildConnection accidentally extended its SIGINT/SIGTERM scope to every connection-opening process. The synchronous process.exit(130/143) in those handlers raced the scheduler-daemon's own async signal handlers — surfaced by the serve lifecycle — bridge failure + shutdown test (its daemon_stopped expectation fires AFTER scheduler-daemon's awaited shutdown flushes to scheduler.log; the synchronous exit short-circuited that flush). Fix: drop the SIGINT/SIGTERM branches; beforeExit/exit still drain the pool on every clean exit, and signal-driven shutdown belongs to the process owner.

Test plan

  • bun run typecheck — clean.
  • bun test src/lib/db.test.ts — 56/56 pass (incl. new G5 regression + lockout tests).
  • bun test src/term-commands/serve.test.ts src/genie-commands/doctor.test.ts src/genie-commands/__tests__/install.test.ts — all pass (pgserve failure containment 3/3, pgserve v1/v2 coexistence 1/1).
  • Full bun test — 4107 pass / 616 skip / 3 fail (the 3 are pre-existing otel-receiver port-collision flakes that pass in isolation and reproduce on origin/dev).
  • bun run check:fast — clean (typecheck + biome + knip + skills/wishes/emit-discipline lints).
  • Wave 1 G1 e2e: PATH=<bun+pm2 only without pgserve> bun src/genie.ts install exits 1 with the canonical install hint.
  • Wave 2 source-grep validation: grep -rn 'selfHealPostgres|startPgserveDaemonOnce|spawnPgserveDirect|terminatePgserveTree' src/ returns only test-side lockout assertions in db.test.ts. grep -rn 'spawn.*pgserve' src/ returns only spawnSync('pgserve', ['install'|'port']) (CLI shellouts, not daemon spawns).
  • Fresh-host docker smoke (deferred to felipe per the engineer-2 brief): docker run --rm -it ubuntu:24.04 bash -c '…pgserve install && genie install && genie doctor'.
  • Live-host pm2 cycle (felipe to coordinate): pm2 stop pgserve && genie serve start should print the pm2-recovery hint within 2s; pm2 start pgserve && genie serve start should print "pgserve daemon ready (canonical, pm2-supervised)".

Migration for pre-canonical operators

# 1. Install canonical pgserve (pm2-supervised singleton)
bun add -g pgserve@^2
pgserve install                # registers under pm2; auto-detects host
# If you have existing data at ~/.genie/data/pgserve and want to keep it,
# point pgserve install at that data dir BEFORE the cutover:
pgserve install --data ~/.genie/data/pgserve

# 2. Re-run genie install (fails fatally if pgserve isn't ready)
genie install

# 3. Verify
genie doctor                   # all [ok] for pgserve preconditions

🤖 Generated with Claude Code

Genie has no embedded pgserve fallback after the canonical-cutover wish.
`genie install` must surface a missing or broken pgserve at install
time, not at runtime.

- tryPgserveInstall → requirePgserveInstall: void on success, exits 1
  with a copy-paste recovery hint on any failure (binary missing, install
  non-zero exit, port discovery failure).
- Hint format: "canonical pgserve registration failed (<reason>)" + the
  three recovery commands (bun add -g pgserve@^2 / pgserve install /
  genie install) + docs/install.md URL.
- Remove warn-and-continue branch and the stale "(it has its own pgserve
  daemon embedded in genie serve)" comment fragment.
- Drop the now-unused `note` helper.
- Tests cover hint shape (5 new cases). E2E fatal-exit verified by
  running `bun src/genie.ts install` against a PATH without pgserve.

Validation:
  PATH=<bun+pm2 only> bun src/genie.ts install   # exits 1 with hint
  bun test src/genie-commands/__tests__/install.test.ts  # 19 pass
  bun run typecheck                              # clean
…+G4)

Genie was a daemon OWNER pre-cutover: getOrStartDaemon would spawn
pgserve via Modes B (recover) + C (spawn), selfHealPostgres pkilled
postgres backends to recover stuck state, and `genie serve start`
treated pgserve startup as part of its boot sequence. Canonical
pgserve@^2 is a pm2-supervised singleton; every pkill of a pm2
process triggered immediate respawn, producing the "Could not kill
stale postgres processes" + "pgserve v2 daemon exited before binding"
fight-with-pm2 cycle that motivated this wish.

This commit converts genie to consumer-only.

G2 — getOrStartDaemon → requirePgserveDaemon:
- Single Mode A: probe + greet via probePgserveDaemon +
  isPgserveSocketResponsive. On success returns DaemonState; on
  failure throws a pm2-recovery hint error.
- Deleted: daemonStartPromise single-flight, cleanPartialDaemonState,
  recoverUnresponsivePgserveDaemon, isLikelyPgserveDaemonProcess,
  signalPgserveDaemonPid, removeStalePgserveSocketArtifacts,
  unlinkIfPresent (only used by deletees).
- getOrStartDaemon kept as a deprecated alias for one release with
  a stderr deprecation notice on first use.
- Call sites updated to requirePgserveDaemon directly.

G3 — Delete spawn helpers + replace serve startPgserve:
- Deleted from db.ts: startPgserveDaemonOnce, evictOrphanDataDirHolder,
  detectOrphanDataDirLock, OrphanDataDirHolder, waitForDaemonSocket,
  formatPgserveDaemonCommand, spawnPgserveDirect, startPgserveOnPort,
  findPgserveBin, findPgserveDaemonCommand, findLocalPgserveRoot,
  resolvePgservePackageCommand, findBunRuntime, signalPgserveTree,
  terminatePgserveTree, waitForDaemonPort, throwDaemonTimeout,
  PgserveDaemonCommand interface, sleep helper, maskCredentials,
  pgserveChild module state, isPgAutostartDisabled, TRUTHY_ENV.
- _ensurePgserve simplified: only force-TCP non-test reaches it; if
  no existing port is reachable, throw with the canonical install
  hint — genie never spawns pgserve.
- registerExitHandler is now invoked once at the end of a successful
  _buildConnection so the postgres pool drain (beforeExit / SIGINT /
  SIGTERM) survives the spawn-helper deletes.
- serve.ts: startPgserve → requirePgserveReady (probe-only). On
  success logs "pgserve daemon ready (canonical, pm2-supervised) on
  <socket>"; on failure prints the pm2-recovery hint, sets
  GENIE_PG_NO_AUTOSTART=1 + GENIE_PG_DISABLE_AUTOSTART=1, exits the
  function so the rest of the serve boot doesn't loop on the same
  failure.
- autoStartDaemon's outcome-tracking variables (lastAutoStartOutcome,
  lastAutoStartPid) deleted — the consumer-only _ensurePgserve no
  longer reads them and the branched-timeout error path is gone.

G4 — Delete selfHealPostgres + doctor pkill:
- selfHealPostgres deleted from db.ts (its only callers were the
  deleted spawn helpers).
- killStalePostgres in doctor.ts replaced with printPgserveRecoveryHint
  (hint-only). doctor --fix never shells out to pkill pgserve/postgres.
  Recovery is the operator running `pm2 status` / `pm2 restart pgserve`
  / `pgserve install` themselves.

Tests:
- src/lib/db.test.ts — replaced ~10 source-text assertions on
  deleted helpers with a single canonical-cutover lockout test that
  asserts every removed symbol is absent. Updated tests on surviving
  surfaces (canCompletePgserveGreet, _buildConnection probe flow,
  hint-message shape).
- src/term-commands/serve.test.ts — pgserve failure containment
  suite rewritten for the consumer-only `requirePgserveReady` flow.
- src/genie-commands/doctor.test.ts — v1/v2 coexistence suite
  rewritten: doctor now never pkills, only prints recovery hints.

Validation:
  bun run typecheck                    # clean
  bun test src/lib/db.test.ts          # 55 pass
  bun test src/term-commands/serve.test.ts src/genie-commands/doctor.test.ts  # all targeted pass

Net change: ~745 LOC removed in db.ts, ~220 LOC added across tests +
new hint helpers. Surface area collapses ~3x.
g5 — regression coverage:
- Adds `requirePgserveDaemon never spawns when daemon is healthy` test
  (db.test.ts). Source-text invariant: the function body and its
  deprecated alias never reference any child_process spawn primitive
  (spawn / spawnSync / execSync / execFileSync) nor process.kill —
  and MUST call probePgserveDaemon + isPgserveSocketResponsive. The
  brief asked for a behavioural mock-spawn test, but Bun's import
  cache makes spy-then-reimport brittle; the source-text assertion is
  strictly stronger because it covers every code path through the
  function, not only the one the mocked test would exercise.

g6 — install docs:
- README.md: added a "Manual install (canonical pgserve first)"
  subsection with the three-step canonical pattern (bun add -g
  pgserve@^2 / pgserve install / bun add -g @automagik/genie /
  genie install / genie doctor) and forward-link to docs/install.md.
- CHANGELOG.md: prepended an Unreleased "pgserve canonical cutover"
  entry that documents the breaking changes (no more spawn, deleted
  symbols, doctor pkill removed, fatal install on failure) and ships
  a copy-paste migration block for pre-canonical operators (including
  `pgserve install --data ~/.genie/data/pgserve` for keeping existing
  data dirs).

regression fix uncovered during wave 3 validation:
- db.ts registerExitHandler had been moved from the (now-deleted)
  spawn-side call site into _buildConnection so the postgres pool
  drain still runs on every connecting process. The function's
  SIGINT/SIGTERM branch synchronously called process.exit(130/143),
  which races the scheduler-daemon's own async signal handlers — the
  failure mode the `serve lifecycle — bridge failure + shutdown` test
  surfaces (the test's daemon_stopped expectation fires AFTER the
  scheduler-daemon's awaited shutdown flushes to scheduler.log; the
  synchronous exit short-circuited that flush). Pre-cutover the
  handlers only fired in OWNER processes, where the race didn't
  matter; post-cutover the helper runs in every connection-opening
  process, which makes the race universal.
- Fix: drop the SIGINT/SIGTERM handlers from registerExitHandler.
  The 'beforeExit' + 'exit' wiring still drains the pool on every
  clean exit; signal-driven shutdown is the responsibility of the
  process owner (scheduler-daemon, TUI), not this consumer-side
  helper. Restores the failing serve-lifecycle test to green.

validation:
  bun run typecheck                                              # clean
  bun test src/lib/db.test.ts src/term-commands/serve.test.ts \
           src/genie-commands/doctor.test.ts \
           src/genie-commands/__tests__/install.test.ts          # 113 pass
  bun test  # 4107 pass / 616 skip / 3 fail (all pre-existing
             # otel-receiver port-collision flakes that pass in
             # isolation; reproduce on origin/dev pre-cutover).
…d-code gate

The pre-cutover wish suggested keeping `getOrStartDaemon` as a deprecated
alias for one release. The project's `dead-code` (knip) gate doesn't
honour `@deprecated` JSDoc nor inline ignore comments, and adding the
symbol to knip.json's allowlist would defeat the gate's purpose. Cleaner
path: remove the alias, document the rename in the CHANGELOG, and let
downstream callers migrate to `requirePgserveDaemon`.

- Removes `getOrStartDaemon` export from db.ts.
- Updates the G5 regression test to slice
  `requirePgserveDaemon` against the next-defined function
  (`buildPgserveUnavailableHint`) and adds a defence-in-depth lockout
  on `export async function getOrStartDaemon`.
- CHANGELOG entry updated to flag the rename as a breaking change with
  the rationale (dead-code gate behaviour) and to direct downstream
  callers to the new symbol.

Validation:
  bun run typecheck                  # clean
  bun test src/lib/db.test.ts        # 56/56 pass
  bun run check:fast                 # all gates green
@coderabbitai

coderabbitai Bot commented May 4, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: bece0640-2ef1-4a2b-82cf-fc0326a93980

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/pgserve-canonical-cutover

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@namastex888
namastex888 merged commit e0196b7 into dev May 4, 2026
16 checks passed

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements the "pgserve canonical cutover," transitioning Genie from a daemon owner that manages its own pgserve instance to a consumer of a pm2-supervised singleton. Key changes include the removal of over 700 lines of process management logic in src/lib/db.ts and the introduction of probe-only reachability checks. Installation and startup processes now treat pgserve availability as a fatal prerequisite. Review feedback identifies an unused spawn import and suggests re-evaluating the state management for environment variables that disable autostart retries.

Comment thread src/lib/db.ts
*/

import { type ChildProcess, execFileSync, execSync, spawn } from 'node:child_process';
import { spawn } from 'node:child_process';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The import of spawn is unused after the removal of the spawn helpers. Please remove it to keep the imports clean.

Suggested change
import { spawn } from 'node:child_process';
import { } from 'node:child_process';

Comment on lines 627 to 628
process.env.GENIE_PG_NO_AUTOSTART = '1';
process.env.GENIE_PG_DISABLE_AUTOSTART = '1';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The environment variables GENIE_PG_NO_AUTOSTART and GENIE_PG_DISABLE_AUTOSTART are set to '1' but never unset or managed in a way that allows recovery without a process restart. Consider if this is intended behavior or if a more robust state management is needed.

@automagik-genie
automagik-genie deleted the fix/pgserve-canonical-cutover branch September 25, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants