feat: add official vLLM Router frontend

[cquil11]

Human written description READ FIRST

Goal: add support for vLLM router in addition to existing support for SGLang model gateway

• add src/srtctl/frontends/static_router.py to encompass logic between "native engine routers" ie vllm and sglang
  • move router discovery code from sglang to static router implementation
  • add vllm_router.py
• frontend: vllm is kept for backwards compatibility, but really this should never be used. before, this meant "launch vllm engine and set up router manually"
• add logic for agg single node, agg multi node, and disagg for vllm router. they are all slightly different
  • agg single node: ie when 4 gpus per node and running dep4 (ep4, dp2, tp2)
  • agg multi node: ie when 4 gpus per node and running dep8 (ep8 dp4/2, tp2/4)
  • disagg: separate prefill/decode engines with a combination of the two above
  • srt-slurm does some verification on combination of dp/tp/pp/dcp according to vllm logic to protect user from giving invalid configs

Summary

Add first-class support for the official vLLM Router while preserving all existing frontend names and behavior:

frontend.type	Behavior
vllm	Existing router-free, single-node aggregate vllm serve
vllm-router	Official vLLM Router over private vllm serve endpoints
sglang	Existing native SGLang Router
dynamo	Existing Dynamo request plane

The implementation introduces a shared static-router abstraction for the orchestration that SGLang Router and vLLM Router have in common, with small adapters for their protocol and CLI differences.

Architecture

Frontend implementations register through @register_frontend(...), and get_frontend() resolves the configured adapter through that registry. Existing frontend types remain registered under their current names; vllm-router is additive.

StaticRouterFrontend owns the common static-router lifecycle:

• collect logical worker leaders and preserve aggregate, prefill, and decode roles;
• reject empty, incomplete, or mixed aggregate/P-D topology;
• build aggregate or role-aware P/D Router arguments;
• launch critical Router processes through the existing frontend/nginx placement;
• merge recipe and runtime environments and run the configured setup script;
• expose Router worker health through the frontend protocol.

SGLangFrontend and VLLMRouterFrontend provide only the backend-specific command, URL, health, and bootstrap behavior.

vLLM Router behavior

• frontend.type: vllm-router requires backend.type: vllm.
• Aggregate workers are advertised with --worker-urls.
• Disaggregated workers use --vllm-pd-disaggregation, repeated --prefill and --decode endpoints, and the allocated prefill NIXL bootstrap ports.
• X-Session-ID can be passed to Router policies such as consistent_hash for stable AgentX conversation affinity.
• Router's worker-startup timeout defaults to the srtctl health-check window; an explicit recipe value wins.
• Router may use its own frontend.container_image, including cluster image aliases.
• Router stdout and stderr are captured in the normal per-process server-log artifact bundle.

Direct frontend.type: vllm is unchanged: it launches one aggregate vllm serve on the public inference port with no Router process.

vLLM parallel topology

Direct vLLM and single-node vLLM Router workers preserve native vLLM behavior: one vllm serve command owns the configured DP, TP, PP, and PCP ranks and performs vLLM's internal DP load balancing.

For multi-node Router DP, backend.dp_launch_mode: per_node launches one hybrid-LB server per node. srtctl validates and derives the distributed topology from the Slurm allocation:

• allocated GPUs must equal DP * TP * PP * PCP;
• local DP is local GPUs / (TP * PP * PCP);
• global DP start ranks advance by the derived local DP count;
• all routed pools must have the same local DP size because Router has one --intra-node-data-parallel-size value;
• manually setting srtctl-owned DP coordinator, rank, local-size, hybrid-LB, or headless fields is rejected rather than silently overwritten.

The legacy Dynamo per_gpu layout remains available for pure DP with TP=PP=PCP=1. Multi-node TP-only direct serving is rejected explicitly instead of being misinterpreted as DP.

Readiness and observability

The existing Router /workers count is necessary but not sufficient: official vLLM Router can list every DP-expanded rank before every advertised base API is ready. After the expanded-count gate passes, the vLLM Router adapter therefore requires HTTP 200 from every exact backend /health URL before launching the benchmark. Other frontend adapters retain their existing readiness semantics.

AIPerf receives every logical vLLM worker metrics URL. Static/direct frontends skip Dynamo's NATS and etcd infrastructure, and logical endpoint selection avoids duplicate metrics from follower ranks.

Final validation

Current srt-slurm head: 0f2a4ddd67aa17075000826c763085541d38c728

• Current-head CI: lint, typecheck, tests, mock server, and recipe validation passed; copyright and CodeQL also passed.
• Parallel-topology/configuration tests: 184 passed.
• Full local suite: 955 passed; the five remaining local path/platform failures reproduce outside this change.

The final official InferenceX Router sweep is workflow 31502477920, which completed successfully:

Topology	AgentX profile	GSM8K exact match
aggregate DEP4	418/418, 0 errors	0.9636
disaggregated 1P/2D DEP4	511/511, 0 errors	0.9629
disaggregated 2P/2D DEP4	518/518, 0 errors	0.9621
aggregate multinode DEP8	422/422, 0 errors	0.9606

All four performance jobs completed Slurm with exit code 0:0, passed the required vllm: metrics gate, and used the all-base readiness barrier. Every evaluation completed all 1,319 GSM8K requests with eval_exit=0. The workflow published Router logs, backend logs, raw and aggregate AgentX results, evaluation outputs, and metrics artifacts for every topology.

InferenceX integration: SemiAnalysisAI/InferenceX#2549

---

Note

Medium Risk
Touches job startup (health gating, infra skip, vLLM process layout) and routing CLI for multiple frontends; misconfiguration is mostly caught at recipe load, but runtime regressions could affect SGLang and Dynamo vLLM DP paths.

Overview
Adds frontend.type: vllm-router, which runs the official vllm-router process against private vllm serve workers (aggregate --worker-urls, disaggregated --vllm-pd-disaggregation with NIXL bootstrap ports). Direct vllm stays router-free on a single aggregate node; sglang and vllm-router skip NATS/etcd like other static frontends.

Refactors SGLang routing into a shared StaticRouterFrontend (worker URL collection by positive HTTP port, P/D vs agg CLI, optional frontend.container_image). Frontends register via @register_frontend; health parsing goes through each adapter, with check_static_router_health for /workers. Benchmark startup can also wait_for_http_endpoints when an adapter requires every advertised backend /health (vLLM Router readiness race).

vLLM backend changes tighten DP topology: default dp_launch_mode: per_rank (per_gpu alias), validate GPUs = DP × TP × PP × PCP, derive hybrid-LB per_node pools for multi-node Router DP, and build router worker commands with private ports/KV transfer. Load-time validation covers router/backend pairing, SGLang tp-size % dp-size, and conflicting manual DP coordinator flags. SGLang P/D under the native router gets --skip-server-warmup so readiness follows the real router path.

^{Reviewed by Cursor Bugbot for commit d6b2f91. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 328280d. Configure here.}