fix(platform): panic-recovering supervisor for every background goroutine (#92)

[HongmingWang-Rabbit]

Summary

Addresses the systemic bug exposed by #85 (scheduler died silently for 12+ hours): every long-running goroutine in the platform lacked panic recovery and a liveness signal. In SaaS this is a scale-of-one sev-1 — one tenant's bad data panics a subsystem → every tenant loses it → every health probe stays green.

Introduces platform/internal/supervised/ with RunWithRecover + a liveness registry, wraps every go X.Start(ctx) in main.go, adds GET /admin/liveness for operator visibility.

Changes

New package platform/internal/supervised/ (142 LoC + 120 LoC tests):

• RunWithRecover(ctx, name, fn) — recover wrapper with exponential backoff (1s → 2s → 4s → … → 30s cap). Panic → restart. Clean return → stop. ctx.Done → stop.
• Heartbeat(name) / LastTick(name) / Snapshot() / IsHealthy(expected, staleThreshold) — in-memory liveness registry.

platform/cmd/server/main.go — wraps every background goroutine:

Before	After
go broadcaster.Subscribe(ctx)	go supervised.RunWithRecover(ctx, "broadcaster", broadcaster.Subscribe)
go registry.StartLivenessMonitor(ctx, onOffline)	wrapped
go registry.StartHealthSweep(ctx, prov, 15s, onOffline)	wrapped
go cronSched.Start(ctx)	go supervised.RunWithRecover(ctx, "scheduler", cronSched.Start)
go channelMgr.Start(ctx)	wrapped

platform/internal/scheduler/scheduler.go — first subsystem to call supervised.Heartbeat("scheduler") after each tick. Demonstrates the end-to-end pattern; follow-up PRs will add heartbeats to the other four.

platform/internal/router/router.go — new GET /admin/liveness endpoint returning { subsystems: { "<name>": { last_tick_at, seconds_ago } } }.

Tests

platform/internal/supervised/supervised_test.go — four cases:

• Clean return does not restart
• Panic restarts with backoff (deliberately panic 3 times, verify recovery)
• ctx.Done stops the restart loop
• Liveness registry: heartbeat → read back, IsHealthy with fresh/stale thresholds, Snapshot is a copy

Local: ok github.com/Molecule-AI/molecule-monorepo/platform/internal/supervised 4.115s. CI will re-validate.

Why this gates SaaS launch

Per #92: in multi-tenant production, every subsystem outage is simultaneous for every tenant, silent, and invisible to standard health probes (HTTP, container, DB all stay green). A single bad row, malformed cron expression, or oversized A2A payload from ONE tenant kills the subsystem for EVERY tenant. Platform restart doesn't help if the bad input is persistent. That is exactly the failure mode we hit on 2026-04-14.

With this PR: a panic is logged + the subsystem restarts in 1-30s; /admin/liveness surfaces stuck-but-not-crashed subsystems too (e.g. a tick that deadlocks); operators can alert on seconds_ago > 2×interval.

Related

• Platform: scheduler stops firing crons silently — ran for hours then died #85 — scheduler silent outage (the canary). This PR generalises the narrow scheduler recover from fix(scheduler): recover from panics + add liveness watchdog (#85) #90.
• docs(plan): add Phase 32 — Cloud SaaS launch roadmap #59 — Phase 32 Cloud SaaS launch roadmap. This PR is on the critical path.

Follow-ups (separate PRs)

1. Add supervised.Heartbeat calls inside each of the other 4 subsystems (liveness-monitor, health-sweep, broadcaster, channel-manager) so /admin/liveness is fully populated.
2. Gate /health on supervised.IsHealthy([...], 2×interval) so external orchestrators (k8s, fly.io) see 503 when a subsystem is stuck.
3. Prometheus metric supervised_last_tick_seconds_ago{subsystem="..."} for scrape-based alerting.

🤖 Generated with Claude Code