Measure and surface per-server database connection budgets (GH-3397)

[jeremydmiller]

Phase 1 of #3397 — measure and surface only. Wolverine reports database connection pressure; it does not yet react to it. Targets 6.19.0.

Why per-server

Connections are a resource of the database server, not of a logical database. That distinction is invisible when an app talks to one database, and load-bearing in the deployment #3397 comes from: ~512 tenant databases spread over a handful of Postgres clusters, every node drawing on the same finite pool. The reporter's actual failure was a schema-apply job that couldn't get a connection.

So the budget is keyed by server, and — critically — the probe is deduped by server: one query per server per metrics sweep, however many tenant databases the node happens to own. A node owning 200 databases across 2 servers issues 2 probes per sweep, not 200. Probing per database would multiply the very pressure the number exists to reveal. There's a test that pins this.

What's here

DatabaseServerId	Keys the budget by (engine, host, port). DatabaseDescriptor.ServerName is host-only, so two clusters co-hosted on one box would otherwise collide onto one budget. SQL Server leaves Port null — its Data Source already carries the port or named instance.
IConnectionBudgetProbe	Optional store capability. Implemented for PostgreSQL (sum(numbackends)) and SQL Server (sys.dm_exec_connections). Providers without a cheap server-wide count don't implement it and drop out.
ConnectionBudgets	Fluent per-server config on DurabilitySettings.
ConnectionBudgetSnapshot	Published per server per sweep via a new default no-op IWolverineObserver.ConnectionBudget — existing observers (incl. CritterWatch) are unaffected.
OTel	wolverine-database-connection-count / -budget, tagged by server.

opts.Durability.ConnectionBudgets
    .ForServer("pg-shard-a", 5432, maxConnections: 400)
    .ForServer("pg-shard-b", 5432, maxConnections: 200);

The one design decision worth arguing about

The max side is explicit configuration, not the probed max_connections. Behind a pooler (pgBouncer et al.) the server's limit describes what the pooler may open, not what the application is entitled to take — charting utilization against it would be reassuring and wrong. A probed limit is a clearly-labelled fallback when nothing is declared, and Unknown is reported honestly when neither exists. SQL Server without VIEW SERVER STATE degrades to a single warning per server, not a failed metrics sweep.

The used count is deliberately server-wide and includes other applications — that's the point (shared resource), but it's documented so nobody reads it as "connections Wolverine is using." The pooler caveat gets its own section in the docs, since numbackends behind a transaction-pooling pgBouncer means something quite different from what most people will assume.

Activation

On automatically only for the statically-known-multiple-databases shape (Marten's MultiTenantedWithShardedDatabases) — where a per-server number tells you something a per-database count can't. Off for a single database. Declaring a budget is itself an opt-in, and ConnectionBudgets.Enabled forces it either way. Silent entirely when DurabilityMetricsEnabled is false.

Not in scope

The adaptive half (back off polling/concurrency under pressure) is deliberately not here. The knobs #3397 names — SlowPollingTime/FastPollingTime, the #486 concurrency governors — live in JasperFx.Events' DaemonSettings, not Wolverine, so the back-off hooks have to ship in a JasperFx release. And it wants #3376 resolved first, since owned-agent scoping changes the baseline any tuning would work against. Phase 1 is independent of both, and happens to be the instrument that answers the connection-breakdown question still open on #3376 — which is why it ships first.

Workstream P resolved as (b): SQL Server gets the probe plumbing and gauge parity now; activation follows Polecat's tenancy roadmap (it has no sharded-database tenancy today). No Polecat changes needed.

Testing

22 new tests. dotnet build wolverine.slnx -c Release is clean.

• N databases on one server ⇒ 1 probe per pass (the definition-of-done assertion)
• 4 databases on server A + 1 on server B ⇒ both probed on the same cadence
• configured budget beats the server's own limit, which is then never even read
• probed fallback is read exactly once per process, not once per pass
• unknown budget reported honestly; utilization null
• a failing probe publishes nothing and leaves the rest of the sweep running
• the existing 5 persistence_metrics_sweeper tests still pass

Note: CoreTests.Persistence has 20 pre-existing ClaimCheck failures when the group is run together (verified identical on a clean main) — unrelated to this change.

🤖 Generated with Claude Code