support rust sglang server

[rainj-me]

Motivation

RFC #23206

• Migrate pre-scheduler python server to rust
• DP attn related change within incoming PRs
• IO intensive workload with thread-per-core runtime support within incoming PRs
• Standalone (or external) api-server deployment within in coming PRs
• More status update refer the RFC

Modifications

Splite PRs:

Done

• create rust workspace #32014
• init sglang rust server project #32256
• sglang rust server environ fsm error id gen #32240
• support regex that compatible with python re lib however apply more l… #32676
• sglang rust server request message #32242
• sglang rust server sampling message #32343
• sglang rust server egress message #32342
• sglang rust server tokenizer manager, ring and runtime #32358
• update Cargo.lock for the rust sglang-server dependencies #32871 Cargo.lock only 2,997 (lockfile)
• add the rust server tokenizer, detokenizer, and egress modules #32872 tokenizer.rs, detokenizer.rs, tokenizer_manager/egress.rs 976
• add the rust server ingress request validation and api server common types #32873 ingress.rs (implementation), api_server/common.rs, log.rs, openai.rs 888
• add the rust server ingress tests, guard, and submit modules #32874 ingress.rs (tests), guard.rs, submit.rs 900
• add the rust server api frame codec and http server entry #32875 api_server/frame.rs, api_server.rs 962
• add the rust server native api handlers and runtime threads #32876 native_api.rs, runtime/threads.rs 744
• wire the rust server modules into lib, runtime, and tokenizer manager #32877 wiring: lib.rs, runtime.rs, message.rs, tokenizer_manager.rs, environ.rs, error.rs 735

Pending

1. add python foundations for the rust server (env vars, msgpack decode helpers, flatten utils) #32864 python foundations (reused PR, base retargeted) ~285
2. integrate the rust server with the scheduler, entrypoints, and engine #32865 python integration (reused PR) ~700
3. add tests and eval support for the rust server #32866 tests & eval (reused PR) ~234

Landed

• Ingress: pre-tokenized columnar transfer (raw input_ids buffer, msgpack
  scalar header); logprob/hidden request params plumbed; is_health_check flag
  so the scheduler skips probes without occupying a queue slot (rids stay u64).
• Egress: single-frame batch transport (push_batch) — N per-request
  msgpack encodes + FFI crossings collapsed to one; logprobs (output/input,
  top-k, token-ids), hidden states, return_text_in_logprobs.
• Endpoints: /generate (unary + SSE), /server_info, /get_model_info
  (+/model_info), /v1/models, /health + /health_generate.
• Idle: scheduler parks on the ingress ring (RustServerIdleSleeper) instead
  of spinning at 100% CPU; zero added latency (ring wakes on push).
• Scheduler: accept skips detok/meta_info dead-work in rust mode
  (only the fields the Rust egress reads).
• Removed dead OpenAI-parser / standalone / headless-TCP code paths.

TODO / Open

• Scheduler CPU is the throughput ceiling at high batch (e.g. 4096) once
  tok/detok is offloaded — single Python thread; accept /
  add_one_req_ignore_eos / process_batch_result dominate. Needs per-step
  work reduction / further offload.
• OpenAI endpoints (/v1/completions, /v1/chat/completions) — required
  for lm_eval; currently only SGLang-native /generate.
• Extras path loop-free: keep logprobs/hidden as contiguous tensors upstream
  so the egress is a single .tobytes() instead of a per-request flatten.
• meta_info parity gaps: cached_tokens, reasoning_tokens, spec-decode
  stats, etc. not yet emitted by the Rust egress.
• Ingress drain semantics: SGLANG_SCHEDULER_MAX_RECV_PER_POLL=-1 should mean
  "drain all" (matches zmq); rust drain currently caps at 256 — decide/align.
• Runtime/CI: end-to-end tests + accuracy (gsm8k) on the rust path; wire into CI.
• Multi-node / dp_size > 1 support (currently rank-0 embedded only).
• Authz
• TLS
• Benchmark_serving stall with flood of sse frames, need further tuning
• offline Engine API support
• support Embedding request
• recv_from_rpc is never drained in rust-server mode
• custom_logit_processor
• cache_tokens
• parallel sampling
• return_sampling_mask flag
• make msgpack_decode_explained work with BaseBatchReq?
• preferred_sampling_params is not supported

Accuracy Tests

Speed Tests and Profiling

Checklist

• Format your code according to the Format code with pre-commit.
• Add unit tests according to the Run and add unit tests.
• Update documentation according to Write documentations.
• Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed.
• Follow the SGLang code style guidance.

Review and Merge Process

1. Ping Merge Oncalls to start the process. See the PR Merge Process.
2. Get approvals from CODEOWNERS and other reviewers.
3. Trigger CI tests with comments or contact authorized users to do so.
   • Common commands include /tag-and-rerun-ci, /tag-run-ci-label, /rerun-failed-ci
4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

---

CI States

Latest PR Test (Base): ✅ Run #30587617646
Latest PR Test (Extra): ✅ Run #30598381601

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[rainj-me]

Need to fix:

  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler_components/request_receiver.py", line 93, in recv_requests
    recv_reqs = self._broadcast_reqs_across_ranks(recv_reqs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler_components/request_receiver.py", line 209, in _broadcast_reqs_across_ranks
    recv_reqs = broadcast_pyobj(
                ^^^^^^^^^^^^^^^^
  File "/sgl-workspace/sglang/python/sglang/srt/utils/common.py", line 1474, in broadcast_pyobj
    serialized_data = pickle.dumps(data)
                      ^^^^^^^^^^^^^^^^^^
TypeError: cannot pickle 'mappingproxy' object

[ishandhanani]

Reviewed the current head as the landed Rust /generate + streaming and tokenizer/detokenizer core described in the PR and RFC. I intentionally excluded the documented follow-up work: encoder/multimodal, PD/DP routing, metrics/logging, OpenAI/control/auth/API parity, standalone/multiprotocol, metadata, drain alignment, release/CI, and all performance findings.

The unary and SSE happy paths work, cargo fmt --check passes, and all 16 Rust unit tests pass. The latest commit also correctly moves logprob-text decoding from repeated cumulative API snapshots to once per detokenizer delta.

The inline comments are correctness and lifecycle issues reproduced on the current head: the Python fallback cannot launch, landed logprob inputs can terminate the scheduler, bounded handoffs can drop output or leak request state, listener/model readiness is not synchronized, slow streams can truncate while emitting [DONE], and stop/error semantics diverge. These should be fixed before the Rust path is enabled beyond development use.

[alexnails]

Fable review incoming. Its comments LGTM, but resolve those that do not fit.

[sagearc]

Nice to see the rust frontend path taking shape. One thing I found hard to follow is where the protocol contracts are meant to live. Some user-facing openai paths use dynamo_protocols, some build json directly (openai.rs), and the frontend-to-scheduler uses msgpack. Maybe this is a good opportunity to clarify the contract for both internal communication and external callers?

[rainj-me]

Nice to see the rust frontend path taking shape. One thing I found hard to follow is where the protocol contracts are meant to live. Some user-facing openai paths use dynamo_protocols, some build json directly (openai.rs), and the frontend-to-scheduler uses msgpack. Maybe this is a good opportunity to clarify the contract for both internal communication and external callers?

The dynamo_protocols support the reasoning and function call (tool usage). The openai.rs is just a http api wrapper for the chat/completions api base on the dynamo_protocol.

[ishandhanani]

Full current-head rereview at aa6a0d238b against exact base abafe0022e. Recommendation: request changes. I reviewed all 33 changed files and ran live GPU behavior, exact base-vs-head compatibility checks, CPU-only lifecycle/affinity fault injection, A/B/B/A load, and independent correctness/lifecycle/performance passes.

Validation: cargo fmt --check passed; cargo test --all-targets --locked passed 36/36; clippy with -D warnings failed on the large enum variants and one test-only lookup. Live testing used Qwen/Qwen3-0.6B on 1x L40S. Unary generation, normal SSE, prompt logprobs, malformed-logprob isolation, over-context unary 400, /health, and /health_generate passed. The earlier invalid-ingress leak is fixed: 10,000 invalid requests all returned 400 with 3,032 KiB scheduler RSS growth.

The performance A/B/B/A run was inconclusive: current-head runs differed by 13.8%, while one base phase had 16 approximately five-second tails. I am not treating it as evidence of either a win or a regression.

The inline comments below contain the reproduced or source-proven blockers. Explicitly documented future-work items were excluded.

dismiss