fix(api): cap Npgsql pools + split request/session connections (Supabase pooler exhaustion)

[thomasluizon]

What & why

Production Sentry alert (ORBIT-API-9/-A/-B, EMAXCONNSESSION) — the API exhausted the Supabase Supavisor session pooler (pool_size: 15).

Root cause. A single ConnectionStrings:DefaultConnection (session pooler, port 5432) is shared by EF Core and Hangfire with no pool cap. Npgsql's default Maximum Pool Size is 100, and session mode holds each connection for its whole lifetime — so any concurrency bump opens more than 15 session-pinned backends and the pooler rejects the surplus. The 2026-06-23 trigger was overlapping Render deploys (a Dependabot merge burst): the new instance boots — startup migration + 10 hosted-service first-ticks, each grabbing a connection — while the old instance still holds its session connections, briefly doubling demand past 15. The two erroring call sites (GoalDeadlineNotificationService, AccountDeletionService) are exactly those startup ticks.

Verified: single instance today; UseDurableQueue=false so Hangfire isn't live in prod yet; all explicit transactions already execution-strategy-wrapped; Hangfire.PostgreSql 1.21.1.

The fix (Stage 1 — code; safe to deploy alone)

Root-cause guard + connection-topology split, enforced in code so a deploy can't re-break it:

• OrbitConnectionStringFactory + DatabaseConnectionSettings — apply Npgsql Maximum/Minimum Pool Size in code from config (Database:EfMaxPoolSize=10, Database:SessionMaxPoolSize=5 → together within the 15 ceiling). The cap overrides whatever the env connection string contains, so an env typo can never uncap the pool again.
• EF request path → DefaultConnection (intended for the transaction pooler, port 6543, which multiplexes and scales).
• Migrations + design-time factory + Hangfire → SessionConnection (session pooler, 5432; falls back to DefaultConnection). Required because EF's migration lock (LOCK TABLE "__EFMigrationsHistory" ... ACCESS EXCLUSIVE) and Hangfire's session semantics are silently voided under transaction pooling.
• Startup migration runs on a dedicated session-connection DbContext rather than the runtime (transaction-pooler) context.
• Hangfire capped to WorkerCount=2 + 1-min schedule poll (closes the documented gap: the durable queue was wired to DefaultConnection with no worker/pool cap).

Deploying this PR alone bounds total connections under 15 — the incident cannot recur even before the pooler cutover.

Operator steps (required to fully resolve / Stage 2)

These are env/dashboard changes only the operator can make. The code falls back to current behavior until they are set, so this PR is safe to merge first.

1. Immediate buffer (no deploy): Supabase → Database → Connection pooling → raise Pool Size 15 → ~40 (bounded by the compute tier's max_connections; don't max it).

2. Stage 2 cutover (Render env vars, reversible):

• Set ConnectionStrings__SessionConnection = the session pooler string (port 5432):
  Host=aws-1-us-east-1.pooler.supabase.com;Port=5432;Database=postgres;Username=postgres.<ref>;Password=<pw>;SSL Mode=Require
• Change ConnectionStrings__DefaultConnection to the transaction pooler string (port 6543) with No Reset On Close=true:
  Host=aws-1-us-east-1.pooler.supabase.com;Port=6543;Database=postgres;Username=postgres.<ref>;Password=<pw>;SSL Mode=Require;No Reset On Close=true
• Optional tuning (no redeploy): Database__EfMaxPoolSize, Database__SessionMaxPoolSize.

3. Recommended: raise the per-role server timeout so slower queries aren't clipped independent of Npgsql's client timeout:
alter role authenticated set statement_timeout = '10s';

Rollback = revert the env vars (instant) or revert this PR.

Request-path connection-hold review (audited — no extra code needed)

Audited the request-path holds; three candidates are not connection-pool issues (EF releases the connection between operations), so no change is warranted:

• ProcessUserChatCommand background fact-extraction — Task.Run scope, but the AI call holds no DB connection; only the brief SaveChanges does.
• CalendarAutoSyncService — processes its ≤50 users sequentially in one scope/DbContext (awaited foreach), not a 50-way fan-out; one connection at a time.
• Inline email handlers (SendCode/SendSupport/RequestAccountDeletion) — IMemoryCache + email, or a single read that completes before the send; no connection held across the email. This is a request-latency concern, not connection pressure.

The one real hold — DistributedRateLimitService's brief per-request Serializable transaction — is neutralized by the Stage 2 transaction-pooler cutover (the connection is returned on commit, not pinned). Fully removing Postgres from the rate-limit path (move to Redis) is gated on provisioning Redis and tracked separately.

Test plan

• dotnet build Orbit.slnx — 0 errors.
• dotnet test tests/Orbit.Infrastructure.Tests — 1150 pass, incl. 7 new OrbitConnectionStringFactory cases (caps, env-string override, session/default fallback, config defaults, empty).

🤖 Generated with Claude Code

[claude]

One correctness regression to fix before this lands. Missing retry on migration DbContext (WebApplicationExtensions.cs lines 19-21): the new bare UseNpgsql(migrationConnectionString) call drops the EnableRetryOnFailure(3, 5s) that the DI-registered context had before this PR. During the overlapping-deploy window this PR is designed to survive, the session pooler can still emit a transient EMAXCONNSESSION; without retry the migration throws immediately and the new instance crashes at startup, exactly the scenario this fix is meant to prevent. See inline suggestion for the one-liner fix. Everything else is solid: the factory logic, pool-cap enforcement, the session/transaction pooler split, Hangfire worker cap, and the new test suite all look correct.

[claude]

This lands the connection-pool exhaustion fix cleanly. The factory correctly enforces Npgsql client-side pool caps in code (10 EF + 5 session = 15, within the Supavisor ceiling) so no deploy-time connection string can uncap the pool. The session/transaction pooler topology split, the startup migration isolation onto a dedicated session-connection context, and the Hangfire worker/poll cap are all sound. My prior concern — missing EnableRetryOnFailure on the migration context — was addressed in commit ffd3550. The 7 new factory unit tests cover the key behaviors (cap enforcement, env-string override, session/default fallback, absent config, empty string), and SonarQube reports 0 new issues at 96.7% coverage on new code. Good to go.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
96.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud