From 2ca950426471ee9bba0f0cec29a8e4330ae74b91 Mon Sep 17 00:00:00 2001 From: XinyueZhang369 Date: Sat, 3 Oct 2026 10:24:49 +0000 Subject: [PATCH] docs: update vllm-rust-servicer-selection Signed-off-by: XinyueZhang369 --- .../content/getting-started/grpc-workers.md | 34 +++++++++++++++++++ src/lib/content/getting-started/index.md | 3 ++ 2 files changed, 37 insertions(+) diff --git a/src/lib/content/getting-started/grpc-workers.md b/src/lib/content/getting-started/grpc-workers.md index 3070a10..9fa17b9 100644 --- a/src/lib/content/getting-started/grpc-workers.md +++ b/src/lib/content/getting-started/grpc-workers.md @@ -50,6 +50,8 @@ The vLLM, SGLang, TokenSpeed and MLX gRPC servers come from the `smg-grpc-servic --port 50051 ``` + This starts the default Python servicer. Setting `SMG_VLLM_SERVICER_IMPL=rust` in the worker's environment serves the same contract from Rust instead; see [Rust vLLM Servicer](#rust-vllm-servicer). + === "SGLang" ```bash @@ -100,6 +102,38 @@ The vLLM, SGLang, TokenSpeed and MLX gRPC servers come from the `smg-grpc-servic MLX workers are gRPC-only and run on Apple Silicon. They don't support constrained decoding: chat requests with a forced `tool_choice` or a `response_format` get a 400, as do requests with `n` greater than 1 or string `stop` sequences. +### Rust vLLM Servicer + +vLLM gRPC workers can serve the same gRPC contract from a Rust implementation instead of the default Python one. The selection is newer than v1.11.0: it exists on current main but not in the v1.11.0 wheels or engine images. The switch is the `SMG_VLLM_SERVICER_IMPL` environment variable, read by upstream vLLM's gRPC entrypoint before it builds an engine, so the worker command stays the same: + +```bash +SMG_VLLM_SERVICER_IMPL=rust python -m vllm.entrypoints.grpc_server \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --host 0.0.0.0 \ + --port 50051 +``` + +In Rust mode, the vLLM engine runs headless in a spawned child process and the servicer talks to it over a same-host ZMQ connection; Python keeps only the process lifecycle. That ZMQ hop is inside the worker — the gateway still connects over `grpc://`, so this is not the [ZMQ direct backend](zmq-workers.md). The gateway cannot tell the two implementations apart. + +With `smg serve`, select it with `--servicer-impl rust`. The flag defaults to `python`, is valid only with `--backend vllm --connection-mode grpc` (any other backend or connection mode is rejected at startup), and is authoritative: it sets `SMG_VLLM_SERVICER_IMPL=rust` in each worker's environment, and with `--servicer-impl python` it removes an `SMG_VLLM_SERVICER_IMPL` inherited from your shell. + +```bash +smg serve \ + --backend vllm \ + --connection-mode grpc \ + --servicer-impl rust \ + --model meta-llama/Llama-3.1-8B-Instruct \ + --data-parallel-size 2 +``` + +For the selection to take effect, the worker's environment needs: + +- the `smg` wheel, which carries the Rust servicer binding (`smg.servicer.VllmGrpcServer`) +- `smg-grpc-servicer[vllm]`, which carries the switch and the Rust request path +- a vLLM whose gRPC entrypoint consults the switch (`smg_grpc_servicer.vllm.resolve_servicer_impl`) + +Misconfiguration fails loudly instead of silently serving Python: `smg serve --servicer-impl rust` refuses to start when `smg-grpc-servicer` is not importable or the installed vLLM lacks the hook, and when `SMG_VLLM_SERVICER_IMPL=rust` reaches a worker whose vLLM never consulted it, the Python servicer itself refuses to start. + --- ## Connect SMG diff --git a/src/lib/content/getting-started/index.md b/src/lib/content/getting-started/index.md index 4565b76..83fed7a 100644 --- a/src/lib/content/getting-started/index.md +++ b/src/lib/content/getting-started/index.md @@ -167,6 +167,7 @@ To connect vLLM to its engine core directly instead of through gRPC, add `--conn |--------|---------|-------------| | `--backend` | `sglang` | Inference backend: `sglang`, `vllm`, `trtllm`, or `tokenspeed`. The `SMG_DEFAULT_BACKEND` environment variable changes the default | | `--connection-mode` | `grpc` | Worker connection mode: `grpc`, `http`, or `zmq`. TensorRT-LLM supports only `grpc` and TokenSpeed only `zmq`; `zmq` also works with `vllm` | +| `--servicer-impl` | `python` | Implementation of the vLLM gRPC servicer: `python` or `rust`. `rust` requires `--backend vllm --connection-mode grpc` and is newer than v1.11.0; see [Rust vLLM Servicer](grpc-workers.md#rust-vllm-servicer) | | `--host` | `127.0.0.1` | Router host | | `--port` | `8080` | Router port (`30000` with `--backend sglang`; see below) | | `--data-parallel-size`, `--dp-size` | `1` | Number of worker replicas, each on its own GPU slice | @@ -294,6 +295,8 @@ Use these when workers are not started via `smg serve`. Each command starts one --port 8000 ``` + Setting `SMG_VLLM_SERVICER_IMPL=rust` in the gRPC worker's environment serves the same contract from the Rust servicer instead of the default Python one (newer than v1.11.0); see [Rust vLLM Servicer](grpc-workers.md#rust-vllm-servicer). + === "SGLang" SGLang depends on `smg-grpc-servicer`, so gRPC mode works after `pip install sglang`.