exp(dynamo): researcher-ready native sidecar integration on current main

[samsja]

Summary

Research integration branch for testing Prime-RL against Dynamo's native vLLM sidecar path. This ports the combined implementation from #3181 onto current main without modifying Biswa's branches or PRs.

The branch includes:

• Dynamo /v1/rl/workers discovery for direct vLLM admin endpoints and per-engine world sizes
• external-engine rank mapping and native NCCL/NIXL weight-update RPCs
• vLLM 0.26 token-serving alignment
• grouped checkpoint reload fixes
• Dynamo recipes plus a two-GPU local smoke test
• pinned source-build/install tooling for the exact validated vLLM and Dynamo revisions

Pinned researcher stack

Component	Revision
vLLM	biswapanda/vllm@e74fc3f
Dynamo	ai-dynamo/dynamo@fc556d9
Prime-RL base	main@8c1f196dd

Public vLLM 0.26 and Dynamo 1.3.0 wheels do not contain the complete tested stack. The scripts below build the matching vLLM wheel, Dynamo wheels, and dynamo-vllm-sidecar executable. The custom inference dependencies are installed into .venv-dynamo, isolated from Prime-RL's trainer environment.

Hardware and software prerequisites

The local smoke test requires:

• Linux with two CUDA GPUs
• a CUDA development toolkit and compiler compatible with the pinned vLLM source
• Rust, C/C++ build tools, CMake, Clang/libclang, Protobuf compiler, Git, and uv
• etcd, curl, and jq available on PATH
• enough disk space and time for a source vLLM build

GPU assignment in the checked-in runbook:

GPU	Process
GPU 0	custom vLLM inference engine
GPU 1	Prime-RL trainer

Build and install

Run from the root of this PR checkout:

git submodule update --init --recursive
uv sync --group dev

examples/dynamo/scripts/build_vllm_wheel.sh
examples/dynamo/scripts/build_dynamo_artifacts.sh
examples/dynamo/scripts/install_artifacts.sh

The scripts verify the pinned seven-character revisions before building. Expected outputs:

dist/dynamo/vllm-*.whl
dist/dynamo/ai_dynamo-*.whl
dist/dynamo/ai_dynamo_runtime-*.whl
dist/dynamo/dynamo-vllm-sidecar
.venv-dynamo/

Do not replace these artifacts with uv sync --extra dynamo: the released packages do not contain this complete integration.

Start the local stack

Run each command in a separate terminal from the repository root.

Terminal 1: etcd

etcd --data-dir /tmp/etcd-data \
  --listen-client-urls http://0.0.0.0:2379 \
  --advertise-client-urls http://127.0.0.1:2379 \
  --listen-peer-urls http://127.0.0.1:2380

Terminal 2: custom vLLM on GPU 0

CUDA_VISIBLE_DEVICES=0 VLLM_SERVER_DEV_MODE=1 \
  PYTHONPATH="$(pwd)/src" .venv-dynamo/bin/vllm-rs serve Qwen/Qwen3-0.6B \
  --host 0.0.0.0 --port 8002 --grpc-port 50051 \
  --python "$(pwd)/.venv-dynamo/bin/python" -- \
  --worker-extension-cls prime_rl.inference.vllm.worker.nccl.NCCLWeightUpdateWorker

This exposes native inference gRPC on port 50051 and the vLLM HTTP/admin API on port 8002.

Terminal 3: Dynamo frontend

DYN_ENABLE_RL=true DYN_RL_PORT=8001 \
DYN_VLLM_ENABLE_INFERENCE_V1_GENERATE=true \
  .venv-dynamo/bin/python -m dynamo.frontend \
  --http-host 0.0.0.0 --http-port 8000 \
  --namespace dynamo --discovery-backend etcd \
  --request-plane tcp --event-plane zmq --router-min-initial-workers 1

Dynamo serves OpenAI-compatible generation on port 8000 and RL worker discovery on port 8001.

Terminal 4: Dynamo vLLM sidecar

DYN_NAMESPACE=dynamo DYN_DISCOVERY_BACKEND=etcd DYN_ENABLE_RL=true \
VLLM_HTTP_ENDPOINT=http://127.0.0.1:8002 \
  dist/dynamo/dynamo-vllm-sidecar \
  --vllm-endpoint 127.0.0.1:50051 \
  --admin-endpoint http://127.0.0.1:8002 \
  --model-path Qwen/Qwen3-0.6B \
  --namespace dynamo \
  --rl-discovery-model-name Qwen/Qwen3-0.6B

The sidecar routes Dynamo generation to vLLM over gRPC and advertises vLLM's direct admin endpoint to Prime-RL.

Verify the stack

Do not start training until both commands succeed:

curl -fsS http://127.0.0.1:8000/v1/models | jq .
curl -fsS http://127.0.0.1:8001/v1/rl/workers | jq .

The worker snapshot must have:

• protocol_version == 1
• model Qwen/Qwen3-0.6B
• admin_base_url == "http://127.0.0.1:8002"
• no worker error
• total world_size == 1

Optional pre-training generation check:

curl -fsS http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"Return 2+2."}],"max_tokens":32}' | jq .

Run Prime-RL on GPU 1

CUDA_VISIBLE_DEVICES=1 uv run rl @ examples/dynamo/local/rl.toml \
  --output-dir outputs/dynamo-local --clean-output-dir

The config sends rollout generation to http://127.0.0.1:8000/v1, discovers the direct vLLM control endpoint through http://127.0.0.1:8001/v1/rl/workers, and transfers policy weights directly from the trainer to the vLLM worker using NCCL.

Success criteria:

1. four optimizer steps complete;
2. math rewards are produced;
3. policy versions advance after weight updates;
4. pause/update/resume completes without leaving inference paused;
5. generation succeeds after the updated weights are loaded.

The checked-in version of this runbook is examples/dynamo/local/README.md. Larger deployment recipes are under examples/dynamo/.

SLURM-managed Dynamo

Set [dynamo] enabled = true in a multi-node RL config to replace the normal global vllm-router launch with a managed aggregated Dynamo stack. The launcher starts job-local etcd and the Dynamo frontend on inference node zero, plus one native-gRPC vLLM engine and dynamo-vllm-sidecar per local DP rank. It injects the frontend and discovery URLs into the orchestrator, supervises all processes through the existing Slurm job step, and cleans stale Dynamo processes on startup.

Prepare the pinned artifacts once, then submit the checked-in example:

examples/dynamo/scripts/build_vllm_wheel.sh
examples/dynamo/scripts/build_dynamo_artifacts.sh
examples/dynamo/scripts/install_artifacts.sh

uv run rl @ examples/dynamo/slurm-managed/rl.toml \
  --output-dir outputs/dynamo-slurm --clean-output-dir

Render and validate the job without submitting:

uv run rl @ examples/dynamo/slurm-managed/rl.toml \
  --output-dir /tmp/dynamo-slurm --clean-output-dir --dry-run
bash -n /tmp/dynamo-slurm/rl.sbatch

Configuration and scaling details are in examples/dynamo/slurm-managed/README.md. The initial launcher supports aggregated dense inference on dedicated inference nodes with a shared project filesystem. Disaggregated P/D and cross-engine expert parallelism are rejected explicitly.

Validation completed

• GitHub Ruff: passed
• GitHub Unit tests: passed
• GitHub Slim install: passed
• GitHub CodeQL and analysis: passed
• 28 focused Dynamo discovery, auth propagation, topology, NCCL reload, serving, and orchestrator tests passed locally
• 4 focused Dynamo configuration tests passed locally
• local, Qwen3 0.6B, Qwen3 30B, and GLM/R2E Dynamo configs resolve on current schemas
• build scripts pass bash -n; example TOMLs parse successfully
• generated Dynamo multi-node Slurm script passes bash -n

The source artifact builds and full two-GPU run remain the research-host validation step. The larger GLM recipe additionally requires the optional r2e-gym environment package.

[cursor]

NCCL GPU check crashes on None

Medium Severity

validate_enough_devices_for_nccl reads self.weight_broadcast.inference_world_size while weight_broadcast is still optional and this validator runs before auto_setup_weight_broadcast fills it. An RL config that sets trainer NCCL without a top-level weight_broadcast block now raises AttributeError during validation instead of a config error or the previous GPU check.

 

^{Reviewed by Cursor Bugbot for commit e9d3f87. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 33d5cda. Configure here.}

[cursor]

SLURM vLLM launch omits engine split

High Severity

The SLURM Dynamo engine command never inserts the -- splitter that every validated vllm-rs serve invocation uses to separate Rust frontend flags from Python EngineCore flags. --worker-extension-cls, --tensor-parallel-size, --max-model-len, and the generated dynamo_vllm_args are therefore parsed as frontend arguments, so managed engines fail to start and NCCL weight updates cannot attach.

 

^{Reviewed by Cursor Bugbot for commit 33d5cda. Configure here.}