fix(db, supervisor): fix OOM crashes from broken database cleanup and process restart logic

[oyi77]

Summary

Fixes the root cause of OmniRoute's repeated crashes and restarts — a 1.4GB SQLite database caused by silent cleanup failures, combined with process supervisor issues that created cascading EADDRINUSE failures.

Closes #4425.

Root Cause Analysis (Systematic Debugging)

The Smoking Gun: 1.4GB SQLite Database

Profiling the running process revealed:

• Child process (Next.js server): 1.7GB RSS, 14 threads
• SQLite database: 1.4GB (storage.sqlite)
• Top tables: compression_analytics (601K rows, 52 days), usage_history (258K rows), proxy_logs (100K rows)

The cleanup system EXISTS but had three SQL column name bugs that caused ALL retention queries to silently fail:

Table	Cleanup Query	Actual Column	Bug
compression_analytics	WHERE created_at < ?	timestamp	Wrong column name
call_logs	WHERE created_at < ?	timestamp	Wrong column name
memory_entries	DELETE FROM memory_entries	Table is memories	Wrong table name

Each failed query was caught by try/catch, logged as an error, but the cleanup continued reporting 0 deletions. The database grew unbounded for 52+ days.

Why Cleanup Never Ran

runAutoCleanup() was only called from the dashboard UI (user clicks a button). There was no background timer — so on a headless server, cleanup never ran automatically.

The Cascade

1. Database grows to 1.4GB → better-sqlite3 maps it into memory → 1.7GB RSS
2. Process RSS grows to 3-8GB (with V8 heap, TLS, etc.)
3. systemd MemoryMax=6G cgroup limit → kernel kills process (exit code 0)
4. Supervisor treats exit 0 as "intentional" → exits parent
5. Restart=on-failure ignores clean exit → service stays down
6. Manual restart → port 20128 still in TIME_WAIT → EADDRINUSE cascade

Changes

Commit 1: fix(supervisor) — Process stability

bin/cli/runtime/processSupervisor.mjs:

• Port-wait before restart (isPortFree() + waitUntilPortFree())
• Exit code 0 as restartable when not shutting down (OOM cgroup)
• RESTART_RESET_MS: 30s → 60s, maxRestarts: 2 → 3
• RSS memory monitoring (warn at 80%, restart at 90%)
• Signal parameter in handleExit

bin/cli/utils/portCheck.mjs (new): Port availability checker

bin/cli/commands/serve.mjs: Default V8 heap 512MB → 1024MB, port passthrough

Commit 2: fix(db) — Database cleanup (THE ROOT CAUSE)

src/lib/db/cleanup.ts:

• Fix compression_analytics: WHERE created_at → WHERE timestamp
• Fix call_logs: WHERE created_at → WHERE timestamp
• Add cleanupProxyLogs() for the proxy_logs table
• Add proxyLogs to runAutoCleanup() results
• Add startCleanupScheduler() — background timer (startup + every 6h) with VACUUM

src/server-init.ts: Wire startCleanupScheduler() into startup sequence

Systemd service (operator change)

-RestartSec=30
+RestartSec=10
+StartLimitIntervalSec=300
+StartLimitBurst=10

Verification

# All 21 stability tests pass
node --import tsx/esm --test tests/unit/port-check.test.mjs \
  tests/unit/process-supervisor-stability.test.mjs \
  tests/unit/cleanup-column-fix.test.mjs

# Typecheck clean
npm run typecheck:core

Expected Impact

Metric	Before	After
SQLite database size	1.4GB (growing)	<100MB (pruned every 6h)
Process RSS after 9h	3-8GB	~500MB-1GB
Crash frequency	Every 30min-9h	Rare (OOM prevented)
EADDRINUSE cascades	70 in 2 days	0 (port-wait before restart)
Cleanup execution	Manual only (dashboard)	Automatic (startup + 6h)

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[diegosouzapw]

Obrigado pela investigação, @oyi77! 🙏 O núcleo genuíno deste PR — as 5 queries de cleanup com nomes de tabela/coluna errados (call_logs.created_at, mcp_audit_log, a2a_events, memory_entries, compression_analytics.created_at) + o fato de o runAutoCleanup nunca ter sido agendado — é real e valioso, e foi extraído e mergeado na release/v3.8.34 via #4691 (commit autorado por você + crédito no CHANGELOG da 3.8.34).

Não pude mergear este PR como está porque, vindo de uma base antiga, ele carregava mudanças fora de escopo / destrutivas: remoção do startBudgetResetJob() vivo no server-init, um revert do supervisor que desfazia o #4425 já mergeado, uma reversão do file-size baseline, e uma troca não-relacionada do catálogo llm7. O #4691 traz só o core de cleanup (com startBudgetResetJob preservado e um startCleanupScheduler novo que de fato roda a limpeza a cada 6h). Fechando este em favor do #4691. Valeu! 🚀