feat(relay): add detailed readiness metrics

[ravarora2]

Summary

This PR makes relay readiness failures diagnosable without weakening the existing fail-closed readiness contract. It distinguishes Postgres pool acquisition from query execution, Redis pool acquisition, deletion-catalog validation, and the overall two-second deadline; exports a bounded Prometheus contract for rollout dashboards; and fixes the concurrency, shutdown, and listener-boundary semantics needed for those signals to be trustworthy.

Why

The previous /_readiness implementation exposed only an aggregate ready/not-ready result. During a rollout, operators could not tell whether a pod was blocked on:

• acquiring a Postgres connection;
• executing the Postgres readiness query;
• acquiring a Redis connection;
• validating the deletion catalog; or
• the shared readiness deadline.

Adding metrics to the existing handler also exposed three correctness hazards that this PR resolves:

1. the same handler is mounted on both the public application listener and the private Kubernetes health listener, so public requests could otherwise distort rollout telemetry;
2. concurrent probes can finish out of order, allowing an older result to overwrite newer current-state gauges; and
3. a probe started before SIGTERM can finish afterward, incorrectly return 200 ready, and resurrect ready gauges while the process is draining.

Behavior

Readiness evaluation

• Postgres, Redis, and deletion-catalog checks still run under one shared two-second deadline.
• Postgres distinguishes pool acquisition timeout/error from query timeout/error.
• Redis distinguishes pool acquisition timeout/error. This does not claim a Redis command round trip.
• The deletion catalog distinguishes operation timeout/error.
• Multiple failures are reported as multiple_dependencies_failed; exhaustion of the shared deadline is reported as overall_timeout when no more specific completed outcome wins.
• Readiness remains fail-closed: every dependency must succeed for 200 {"status":"ready"}.

Ordered publication and shutdown

ReadinessCoordinator is process-owned and uses one mutex as the linearization point for probe generations, current-state publication, and terminal shutdown.

• Every completed dependency attempt may contribute its truthful counter and duration observation.
• Only the newest admissible probe generation may publish current-state gauges.
• An older, slower probe cannot overwrite a newer probe's gauges.
• begin_shutdown() and probe commit serialize through the same coordinator.
• Once shutdown commits, an in-flight probe cannot return ready or publish ready/current dependency gauges, even if its dependency work later succeeds.

Shutdown without dependency evaluation records only:

• buzz_readiness_checks_total{reason="shutting_down"}; and
• buzz_readiness_state{check="overall"} = 0.

It does not fabricate dependency failures, dependency state changes, or zero-duration latency samples. If shutdown wins after an in-flight evaluation actually ran, those completed dependency attempts may remain as attempt telemetry, but they cannot overwrite shutdown-dominant current state.

Listener and response contract

• The private health listener's /_readiness route is the sole authority for rollout readiness telemetry.
• The public application listener retains /_readiness for compatibility, evaluates the same dependencies, and preserves the existing response shape, but it emits no buzz_readiness_* metrics.
• Ready responses remain 200 {"status":"ready"}.
• Shutdown responses remain 503 {"status":"shutting_down"}.
• Failed private-health responses include the bounded reason plus postgres, redis, and deletion_catalog booleans.
• Failed public compatibility responses retain the dependency booleans but omit the new detailed reason.
• No header, query parameter, path value, or other request-controlled value becomes a metric label.

Prometheus contract

The final schema is intentionally capped at 99 raw Prometheus series per pod.

Metric	Type	Labels	Raw series/pod
buzz_readiness_checks_total	counter	reason	12
buzz_readiness_dependency_checks_total	counter	dependency, typed outcome	11
buzz_readiness_check_duration_seconds	histogram	check	72
buzz_readiness_state	gauge	check	4
Total			99

Closed label sets

reason:

ready
shutting_down
postgres_pool_timeout
postgres_pool_error
postgres_query_timeout
postgres_query_error
redis_pool_timeout
redis_pool_error
deletion_catalog_timeout
deletion_catalog_error
overall_timeout
multiple_dependencies_failed

Valid dependency / outcome pairs are enforced by typed enums:

• postgres: success | pool_timeout | pool_error | operation_timeout | operation_error
• redis: success | pool_timeout | pool_error
• deletion_catalog: success | operation_timeout | operation_error

check is overall | postgres | redis | deletion_catalog.

The readiness histogram has 15 configured buckets concentrated around the two-second deadline:

0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5,
0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.5, +Inf

The contract deliberately removes the redundant overall result label and the histogram outcome label. Pod, ReplicaSet, version, rollout, raw error, SQL, URL, tenant, user, community, pubkey, and request-controlled values are prohibited application labels; infrastructure enrichment can supply deployment identity outside the application metric.

Postgres production seam and CI

The real Db::readiness_check path now supports deterministic production-seam testing while preserving the same acquisition/query implementation used by the relay. The isolated PostgreSQL lane automatically discovers and executes three ignored integration tests covering:

• a held sole connection causing pool timeout, followed by recovery after release;
• closed-pool acquisition error;
• acquisition success followed by query timeout;
• classified query error;
• cancellation while waiting for a connection;
• cancellation during an in-flight query; and
• eventual pool recovery with waiter/in-flight state balanced.

This prevents the central SQLx/Postgres behavior from being merely compiled but never executed in CI.

Tests and verification

• cargo fmt --all -- --check
• ./scripts/test-postgres-test-discovery.sh
• complete cargo test -p buzz-db
• focused cargo test -p buzz-relay readiness
• real-router cargo test -p buzz-relay real_health_route_
• controlled PostgreSQL execution of all three ignored readiness tests
• production health router -> real GET /_readiness -> Prometheus render assertions
• public-router requests proving zero rollout telemetry
• deterministic out-of-order A/B probe tests
• SIGTERM-during-probe tests proving shutdown dominance
• exported metric name/type/exact-label/bucket assertions
• exported raw-series allowlist and 99-series ceiling assertion

The route-to-scrape regression test uses the production Prometheus builder and verifies the 2, 2.5, and +Inf readiness buckets. It fails if the health recording call, health route, public/health boundary, generation fence, shutdown fence, or bucket override is removed.

Risk assessment

Medium. This changes the live readiness publication path and adds synchronization around probe commit/shutdown. The risk is bounded by:

• preserving the existing dependencies and shared two-second deadline;
• keeping readiness fail-closed;
• using a short, process-local mutex only at begin/commit linearization points, not across dependency awaits;
• retaining the public compatibility endpoint while isolating its telemetry;
• enforcing typed, low-cardinality labels and an exported series ceiling; and
• covering the production router, Prometheus exposition, real PostgreSQL seam, concurrency ordering, and shutdown races.

Operational notes

• Dashboards should treat buzz_readiness_state as the latest sampled current state, not as an event stream.
• shutting_down should be excluded from dependency-failure alerts because no dependency failure is implied.
• Do not use default_zero() for missing current-state data; missing/stale is unknown, not healthy.
• Use histogram buckets, heatmaps, max, or average until the Datadog distribution metadata confirms percentiles are enabled; do not title a widget p95 before that live readback.

Non-goals

This PR does not add the broader process-startup lifecycle, worker/listener supervision, shutdown coordinator, WebSocket/huddle handoff, client recovery telemetry, dashboard mutations, Datadog configuration changes, or deployment changes. Those remain separate follow-up work.

References

• Staging dev relay image runbook
• Readiness telemetry contract: deploy/charts/buzz/README.md

[github-actions]

🔐 Codex Security Review

Status: review required for the current range.

The current range is 5aed49b505a7e27f3b0e34dafa53d6c4e8cdcd64...86a7a194a7c282fbfea518d7c8616fa840293434.
A new review must complete for this exact range. When manual authorization
is required, a Block organization member must comment exactly
@buzz-security-review 86a7a194a7c282fbfea518d7c8616fa840293434 to authorize a new review.
Any previous review applies only to its recorded range.

[wpfleger96]

🤖 Combined review verdict from the coordinated three-lane review (two independent source reviews + live E2E verification), at head 86a7a194a.

Approved. No CRITICAL or IMPORTANT findings in any lane.

• Both source lanes independently verified the generation fence, shutdown dominance, the public/private telemetry boundary, the fail-closed shared deadline, the typed bounded label sets, and the 99-series ceiling proof through the production router and recorder. The AppState::begin_shutdown() consolidation leaves exactly one writer of the shutting_down atomic, keeping the readiness fence, WebSocket-upgrade refusal, and mesh drain consistent.
• The three ignored Postgres readiness tests were confirmed discovered and passing in the isolated PostgreSQL CI lane at this head (384/384), covering pool exhaustion/recovery, closed-pool, query timeout/error classification, and cancellation balance.
• Live E2E against the built relay binary with dedicated Postgres/Redis confirmed the ready/not-ready/shutdown response contracts, zero public-listener telemetry (including under forged request-controlled values), correct reason labeling across dependency failures, SIGTERM dominance over an in-flight probe without gauge resurrection, and the documented histogram buckets.

Two non-blocking notes, no action required for merge:

1. PublicationState::shutdown_generation stores a u64 that is only ever checked with is_some(); the generation value computed in begin_shutdown() is never compared. A plain bool would say what it means.
2. The public /_readiness compatibility route still runs the full dependency evaluation for any unauthenticated caller — unchanged from main, noted only as pre-existing.