feat(grpc): expose follower metadata and node-local KV sources by jain-ria · Pull Request #39659 · sgl-project/sglang

jain-ria · 2026-09-15T22:11:22Z

Summary

Expose node-local engine metadata to externally managed sidecars. With native --grpc-port configured, follower nodes start a lightweight server implementing the existing SglangService/GetServerInfo RPC. They do not initialize a tokenizer, inference bridge, or request-serving frontend; other RPCs return UNIMPLEMENTED.

Leader and follower gRPC servers share the same metadata builder. No protobuf or RPC-path changes are required.

Metadata

Existing server-info fields supply node/DP topology, disaggregation configuration, and the shared dist_init_addr for engine-group correlation.
New kv_event_sources lists only this node's actual publishers: global DP rank, exact subscription endpoint, topic, logical block size (including DCP), and optional replay endpoint.
Sources come from publisher sockets and scheduler-ready messages, aggregated across local schedulers. A node with no subscribable publisher reports [].
Wildcard TCP binds advertise loopback addresses for a colocated consumer; bound IPC endpoints are supported. Connect-style/inproc publishers retain their behavior but are not advertised as local subscription sources.
The existing global kv_events descriptor remains unchanged for compatibility. Consumers of node-local discovery use kv_event_sources instead of inferring ownership from global DP size.

Scope

This supersedes the earlier managed-sidecar proposal in this PR. Removed --sidecar-scope, SGLANG_SIDECAR_CONTEXT, follower sidecar launching, and sidecar watchdog changes. Existing SGLang-managed sidecar launch behavior is unchanged.

Only metadata discovery and its server lifecycle are included. KV events still use ZMQ. Dynamo integration, SGLang-managed follower sidecars, metrics/FPM, and live/elastic source membership are follow-ups. Follower metadata is a startup snapshot after scheduler readiness.

Validation

Base: 406c9c71d833570aa89bc26ccb2e9d1518cd4564. The results below were recorded at SGLang c6c49ac1d2721b93becdfc2be7cab46ddf58c0e8.

Focused Python suites: 315 passed, 100 subtests passed (metadata/lifecycle, gRPC bridge, server-info compatibility, arguments, KV publishers, DP controller).
Native Rust tests: 14 passed, including real gRPC metadata roundtrip, rejection of inference/control RPCs, and shutdown.
Python/native integration smoke: fetch metadata over gRPC, subscribe to its actual ZMQ source, receive a KV event, verify duplicate-port failure and clean/idempotent shutdown.
Rust Clippy all-targets with -D warnings, Rust formatting, and scoped Python formatting/lint passed.
Physical two-node Dynamo integration (2026-09-16): one H100 per node, Qwen/Qwen3-0.6B, NNODES=2, TP_SIZE=2, attention DP_SIZE=2, aggregated serving. Verified generation on both ranks, follower metadata discovery, actual KV Stored events from global DP ranks 0 and 1, one serving registration with two event sources, and an unpinned repeated prompt routed to the follower with 256 cached tokens.
- Tested with Dynamo 0d41b47c28b0a092671f73e3db5bd5d4908ee8f5 from ai-dynamo/dynamo#14908. Both native extensions were rebuilt from the tested sources, using lmsysorg/sglang:v0.5.19-cu130-runtime and a runtime-only sglang-kernel==0.4.7 upgrade.
- The original Dynamo launcher failed to discover a local source. The passing run used an explicit wildcard publisher override, --kv-events-config '{"publisher":"zmq","endpoint":"tcp://*:5557","topic":""}'. Follower Dynamo health still reported 503 despite working event delivery; downstream launcher/readiness fixes were made afterward. This was not an entirely healthy deployment run.

The two-node GPU run has not been repeated after the later loopback-publisher fix in SGLang c50b251da8e455dfbb594f8c3b7ccae6eca72129. It does not cover prefill/decode disaggregation, NIXL transfers, multimodal inference, independent restart/replay, elastic membership, or performance.