feat(load-reporter): add push-based engine load reporting

[Bakerjc-bgner]

Motivation

Load-aware routing needs one authoritative view of scheduler load across
data-parallel ranks. Polling every Worker from every Router adds fan-out,
creates inconsistent views, and churns connections.

This PR adds the Worker-side reporter: a push channel built on the scheduler's
existing snapshot publication. One shared pull per fire is broadcast to every
registered Router over long-lived bidirectional gRPC streams. Router-side
ingestion, discovery, retries, and the Prefill throughput extension are
intentionally split into follow-up PRs so each layer can be reviewed
independently.

The push channel controls delivery, not snapshot age. End-to-end
freshness stays bounded by the scheduler's snapshot publication interval;
see Freshness.

Modifications

Architecture

flowchart LR
    Scheduler["Scheduler<br/>published per-DP-rank load snapshots"]
    Source["Per-mode snapshot source<br/>manager get_loads / Router SHM reader"]
    Fire["Single fire loop<br/>one pull per fire + rank-set validation"]
    Builder["LoadReport builder<br/>HEALTHY / STALE / UNREACHABLE"]
    Service["Worker gRPC service<br/>--load-reporter-port"]
    Router["Routers dial in<br/>drive Monitor streams"]

    Scheduler --> Source --> Fire --> Builder --> Service --> Router

Loading

Protocol and configuration

• The Worker hosts
  sglang.router.loadmonitor.v1.LoadMonitorService.Monitor(stream RouterFrame) returns (stream WorkerFrame) on a fixed, opt-in --load-reporter-port; each
  external Router dials in and drives its own stream (register → ack →
  bounded first fire → periodic reports). Stable field numbers and enum values
  are preserved.
• RouterFrame = register | update_config | keep_alive | stop, with
  register required first; WorkerFrame = registered | report | error.
  Re-registering the same router_id replaces its old session; different
  router_ids coexist.
• Package grpcio and protobuf through the optional load-reporter extra
  across all supported Python wheel variants.
• Reporting stays opt-in so normal SGLang installations gain no mandatory
  runtime dependencies.

Reporter runtime — one timer, one pull, broadcast

• A single fire-loop task owns the only timer; sessions are passive bookkeeping
  (deadline, lease, capacity-1 queue) and own no task or timer.
• Each fire performs exactly one bounded snapshot pull (get_loads(), 1 s
  budget). The same budget covers one retry when the expected DP-rank set
  changes mid-pull.
• One report per pull, validated and built from that attempt only: HEALTHY,
  STALE, or UNREACHABLE (empty ranks + last_error). There is no persistent
  store and no cross-report merge, so a slow or stale rank is never replaced
  with historical data.
• The fire broadcasts the same report to every session whose deadline fired;
  sessions registered before the next fire share its pull for their initial
  report.
• Each session has a capacity-1 latest-wins queue, lease TTL with keep-alive
  renewal, mid-stream config updates, registration-anchored deadlines, and
  idempotent shutdown.
• Pull, transport, and shutdown failures never propagate into the inference
  request path.

Freshness

End-to-end snapshot freshness is bounded by the scheduler's snapshot
publication path (load_snapshot_publish_interval), not by the gRPC delivery
cadence. The report interval controls how often a report is sent, not how old
the snapshot inside it is — a poll at the same cadence observes data of the
same age.

What the push stream adds over polling is delivery, not freshness: a negotiated
per-Router cadence, connection reuse, one-pull multi-Router fan-out, leases,
and bounded backpressure.

Tokenizer integration

• Single-tokenizer HTTP and native gRPC start one reporter runtime in the
  FastAPI lifespan (native gRPC reuses the same process and listener).
• Multi-tokenizer mode: MultiTokenizerRouter is the sole runtime owner; HTTP
  workers open no streams, and elastic scale changes update the expected
  DP-rank set (observed by the next fire).
• Standalone SMG gRPC starts the reporter at request-manager readiness; older
  servicers without the readiness hook fail loudly when the reporter is
  enabled.
• The embedded offline Engine API does not own the reporter; cleanup runs in
  the HTTP-server lifespan, the SMG gRPC-server finally path, and
  MultiTokenizerRouter.close().

Tests and documentation

• Unit/integration coverage (CPU, real in-process grpc.aio): proto contract,
  fire loop (shared pull deadlines, broadcast to due sessions, per-session
  lease gating, rank-set-change retry sharing one budget, registration-anchored
  cadence, stale/error reports, re-registration, shutdown), snapshot validation
  and report-builder contracts, snapshot sources, service handshake and error
  frames, composition-root lifecycle, and standalone SMG wiring.
• Negative invariants: request dispatch/completion/abort never trigger a pull;
  topology updates never pull immediately.
• Registered GPU E2E tests with a real grpc.aio fake Router cover
  single-owner, multi-owner, and standalone SMG reporting.
• Documentation covers the wire contract, serving-mode matrix, fire-loop
  schedule, freshness model, network/deployment assumptions, and shutdown
  behavior.

Design advantages

• Zero request-path coupling: dispatch, completion, and abort have no edge
  into the reporting graph; there is no per-request overhead.
• Efficient fan-out: one pull per fire shared by all due sessions,
  persistent bidi streams, no Router-side polling fan-out or connection churn.
• Topology-safe: reports preserve per-DP-rank detail; rank-set changes are
  validated and retried within the same fire budget.
• Resilient: per-attempt status semantics, session replacement, leases,
  latest-wins backpressure, and bounded shutdown prevent stale tasks from
  accumulating.
• Extensible: a stable protocol and a single validation entry point provide
  a clean foundation for the separately reviewed Router monitor and future
  metrics.

Accuracy Tests

N/A for model accuracy. This PR does not change model forward code, kernels,
sampling, or generated outputs.

Functional validation:

• Registered CPU tests cover the protocol contract, fire-loop
  scheduling/broadcast, snapshot validation, report builder, service behavior,
  lifecycle, and standalone SMG wiring.
• Registered GPU E2E tests drive a real grpc.aio fake Router against
  single-owner, multi-owner, and standalone SMG modes.
• Generated protobuf/gRPC bindings import successfully and expose the expected
  sglang.router.loadmonitor.v1 service.

Speed Tests and Profiling

N/A for inference throughput benchmarking. This is an opt-in control-plane
feature and does not modify model execution.

The reporter adds no request-path overhead: there are no hooks on the
dispatch/completion paths, and sampling/network I/O runs in the single
background fire task. No inference-speed improvement is claimed. Reporter
cadence and freshness are exercised functionally by the registered tests; no
new micro-benchmarks are claimed in this revision.

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci.
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ❌ Run #34449344106
Latest PR Test (Extra): ❌ Run #34449343918
Latest PR Test (AMD ROCm 10): ❌ Run #34449344253

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[Bakerjc-bgner]

@hzh0425 and @whybeyoung
This branch now includes the complete Engine Load Report feature.

Since the new code changes are quite extensive, I've split them into separate commits by module to make it easier to review. Could you please help arrange a code review? Thanks! 🙏

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[whybeyoung]

please resolve the conflicts.

[SSunsh1ne]

pls resolve the conflict
done

[hzh0425]

@Bakerjc-bgner some ci can't pass, seems this patch break something

pls take a look https://github.com/sgl-project/sglang/actions/runs/32443308833/job/96658282007#step:15:109