[Metrics] Count KV bootstrap and transfer timeout observations

[WyattJia]

Motivation

The existing bootstrap and transfer failure counters do not distinguish deadline expiry from other failure causes. Repeated polling and warning-once logs also make log counts unsuitable for counting timeout observations.

This is reproducible without a model: give a sender or receiver a 10-second deadline, set its existing start time to 20 seconds before the current time, and call the shared timeout helper repeatedly. The request becomes Failed, but current main records no classified timeout observation. The initial regression suite on main ab9d8f029c12cda8167f51d46664d643c8771a6e failed eight count assertions across the two paths.

Modifications

• Add sglang:kv_transfer_timeouts_total{stage="bootstrap"|"transfer"} with two zero-initialized stages.
• Observe expiry in the existing shared bootstrap and waiting-timeout helpers, once per sender/receiver object. Preserve the original deadline clocks, failure recording, terminal status, cache invalidation, and abort delivery.
• Create the counter only for metrics-enabled KV managers. Import Prometheus lazily after launcher initialization and reuse the counter across manager lifecycles within one process.
• Add CPU tests for repeated polling, deadline boundaries, deferred bootstrap timing, cancellation, terminal polling, disabled metrics, room reuse, and existing failure records. Add real Prometheus subprocess export and fresh-directory tests.
• Document that these are rank-local observer events, not unique failed requests or a failure-rate numerator.

This does not introduce new deadlines or count transport errors, session recovery, cancellation, or stalled transfers outside the existing shared timeout helpers. Existing phase failure metrics are preserved.

Related work: #41140 is an open lifecycle consolidation that touches the same helpers but does not add classified timeout counters. #34132 adds request outcomes and broad disaggregation abort reasons; it measures a different unit. #34692 already added the missing NIXL bootstrap timeout. This patch instruments that existing behavior.

Production observations

This proposal extracts two observations from a broader internal instrument already running in a disaggregated deployment with four Prefill workers and two Decode workers. On 2026-10-04 UTC, a read-only check confirmed the instrumentation in all six engine containers and 54 fresh event series: nine initialized event categories per worker. The oldest source sample at the snapshot was 16.5 seconds old.

The deployed counter snapshot contained 5 bootstrap-timeout observations, 23 abort observations, and 0 transfer-timeout observations. Over 2026-09-27 14:52:24 to 2026-10-04 14:52:24 UTC, Prometheus increase() estimates were approximately 5, 23, and 0 respectively, with no counter resets observed for the queried series. These are process/rank observer events, not unique failed requests. The distinction is useful because treating every cancellation as a timeout would obscure the actual deadline-expiry signal.

The deployed internal metric is sglang:kv_failure_events_total{event}. This PR deliberately retains only deadline expiry, names it sglang:kv_transfer_timeouts_total{stage}, and excludes cancellation and the other internal categories. The renamed minimal collector and its current-main adaptations have not been deployed in this environment. The production observations support the need for classification; they do not establish a before/after reduction in failures or validate the transfer-timeout path on hardware. No inference request, fault injection, traffic change, or deployment was performed for this check.

Validation

Rebased onto main affa261e3d289fe4f907c9b2e8d773fef0d36dba and reran the focused tests and changed-file checks.

macOS arm64, Python 3.12.12, PyTorch 2.14.1, Prometheus client 0.26.0:

PYTHONPATH=python python -m pytest \
  test/registered/unit/disaggregation/test_kv_transfer_timeout_metrics.py \
  test/registered/unit/observability/test_kv_transfer_metrics.py \
  test/registered/unit/disaggregation/test_receiver_connection_pool.py \
  test/registered/unit/disaggregation/test_kv_transfer_replica_metric.py \
  -v -p no:cacheprovider
# 20 passed, 14 subtests passed

The two new registered files also pass when executed directly, as CPU CI executes them: 8 timeout tests and 4 collector tests. Changed-file pre-commit checks, Ruff, Python compilation, and git diff --check pass. Mintlify 4.2.559 build validation and broken-link checks pass using Node 24.

The timeout tests call the real shared timeout, failure-recording, and status-update methods with controlled clocks and mocked I/O boundaries. Multiprocess checks use actual child interpreters, Prometheus counters, and MultiProcessCollector; they do not exercise a running SGLang server. macOS uses SGLang's own platform compatibility stubs. The full unit suite, Linux CI, server-level metrics export, GPU, multi-node, and RDMA tests were not run.

Accuracy Tests

Not applicable. The change adds observations to existing failure paths and does not change model execution or outputs.

Speed Tests and Profiling

No inference benchmark was run. Successful timeout checks return before the new observation logic. Counter increments occur on the first observed expiry of each object.

Checklist

• Format your code according to Format code with pre-commit.
• Add unit tests according to Run and add unit tests. Registered CPU regressions passed.
• Update documentation according to Write documentations. The production metrics reference documents the new counter and its counting unit.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed. Not applicable to model output; no GPU or performance benchmark was run.
• Follow the SGLang code style guidance.

Maintainer review and CI authorization remain pending.

---

CI States

Latest PR Test (Base): ❌ Run #37211546738
Latest PR Test (Extra): ❌ Run #37211546250
Latest PR Test (AMD ROCm 10): ❌ Run #37211546498