feat(observability): diagnose blocked-loop vs pool-exhaustion on stalled /health

[nicoloboschi]

Problem

The API and worker run the /health handler and all task work on a single asyncio event loop, and /health acquires a database connection. So a failing Kubernetes liveness probe has two very different root causes that are currently indistinguishable:

1. The event loop is blocked by synchronous work on the loop (CPU-bound parsing, a mis-offloaded SDK call, a third-party lib doing inline work). /health can't be scheduled → probe fails. A restart helps.
2. The connection pool is exhausted — /health's own SELECT 1 can't acquire a connection and blocks up to the acquire timeout, with the loop completely idle. A restart just thrashes and can make it worse.

Today both look identical from the outside: pod restarted, cause unknown. This PR makes the failure self-diagnosing instead of working around it.

What this adds

Two always-on, cheap signals:

1. Event-loop stall watchdog (hindsight_api/loop_watchdog.py)

Runs in a separate OS thread — deliberately, because a coroutine-based monitor would be frozen by the very stall it's trying to observe. It pings the loop and, when the loop fails to service the ping within a threshold, logs the loop thread's stack (naming the exact blocking frame) and emits hindsight.event_loop.stalls / hindsight.event_loop.stall_duration. Works with uvloop (unlike monkeypatch-based blocking detectors). Never raises, never touches loop work. Wired into the worker CLI and the API lifespan.

2. DB pool acquire instrumentation (engine/db/pool_instrumentation.py)

asyncpg exposes pool size/idle but not how many callers are queued waiting for a connection — the signal that actually distinguishes exhaustion from a busy-but-healthy pool. This tracks that (hindsight.db.pool.waiting gauge), records an acquire-wait histogram, and logs a warning with pool stats when an acquire waits too long. Wired into both the PostgreSQL and Oracle backends. health_check() now reports db_acquire_ms and pool utilization in its response payload, so a slow/failing /health is diagnosable from the response alone.

At the failing moment you now get one of:

• EVENT LOOP BLOCKED for 2.1s ... <stack naming the frame> → liveness problem, restart helps.
• slow DB pool acquire: waited 30.0s ... (in_use=10 max=10 idle=0 waiting=7) → readiness problem, restart thrashes it.

Config (static, server-level; enabled by default)

Env var	Default
HINDSIGHT_API_LOOP_WATCHDOG_ENABLED	true
HINDSIGHT_API_LOOP_WATCHDOG_STALL_THRESHOLD_MS	1000
HINDSIGHT_API_LOOP_WATCHDOG_POLL_INTERVAL_MS	250
HINDSIGHT_API_DB_ACQUIRE_WARN_THRESHOLD_MS	1000

Testing

• tests/test_loop_watchdog.py — detects on-loop blocks (and captures the culprit stack), ignores off-loop (threaded) work, stays quiet on a responsive loop.
• tests/test_pool_instrumentation.py — waiter counting through success / mid-acquire / acquire-failure, and slow-acquire logging with pool stats.
• Validated end-to-end against a real pg pool: under a saturated max_size=1 pool, a second acquirer registered waiting=1 and logged the slow-acquire warning with real stats.

Context

Came out of a report (@carter, vectorize-io) of /health hanging in a Bedrock-heavy worker. Investigation showed litellm already offloads boto3 credential resolution to a thread (so that specific cause didn't reproduce), which is exactly why runtime visibility — not a workaround — is the right fix: whatever blocks the loop or exhausts the pool next will now name itself in the logs.