feat: dynamo prime-rl integration

examples/dynamo/README.md

@@ -0,0 +1,80 @@
+# Dynamo native-gRPC deployment requirements
+
+The recipes in this directory use Dynamo for inference and Prime only for the
+trainer/orchestrator. Start from Dynamo's `sidecar_agg.yaml` or
+`sidecar_disagg.yaml`, then apply the following RL overlay. The stock manifests
+are serving examples and do not enable Prime worker discovery or vLLM's admin
+control routes by themselves.
+
+## Frontend
+
+Set these variables on the Dynamo frontend and expose both container ports:
+
+```yaml
+env:
+  - name: DYN_ENABLE_RL
+    value: "true"
+  - name: DYN_RL_PORT
+    value: "8001"
+ports:
+  - name: http
+    containerPort: 8000
+  - name: rl-discovery
+    containerPort: 8001
+```
+
+The frontend Kubernetes Service must also map ports 8000 and 8001. Prime's
+`base_url` targets 8000; `dynamo_discovery_url` targets 8001.
+
+## Every vLLM engine and sidecar pair
+
+The engine HTTP address published by discovery must be reachable from the
+trainer, so bind vLLM to the pod network rather than loopback:
+
+```text
+vllm-rs serve <model> --host 0.0.0.0 --port 8000 --grpc-port 50051 -- \
+  --worker-extension-cls prime_rl.inference.vllm.worker.nccl.NCCLWeightUpdateWorker \
+  <other Python EngineCore arguments>
+```
+
+Install the matching Prime source in the engine image so Python can import the
+worker extension. Set this environment variable on the `vllm-engine`
+container to expose `/pause`, `/resume`, and `/collective_rpc`:
+
+```yaml
+env:
+  - name: VLLM_SERVER_DEV_MODE
+    value: "1"
+```
+
+Set the following variables on each `dynamo-vllm-sidecar` container. `POD_IP`
+must appear before `VLLM_HTTP_ENDPOINT` so Kubernetes expands it:
+
+```yaml
+env:
+  - name: POD_IP
+    valueFrom:
+      fieldRef:
+        fieldPath: status.podIP
+  - name: VLLM_HTTP_ENDPOINT
+    value: "http://$(POD_IP):8000"
+  - name: DYN_ENABLE_RL
+    value: "true"
+```
+
+Keep the existing `--grpc-endpoint 127.0.0.1:50051`: gRPC stays pod-local,
+while discovery publishes the pod-reachable HTTP admin address. No per-worker
+Kubernetes Service is required when trainer-to-pod networking is routable.
+
+After startup, `/v1/rl/workers` must return every expected worker with a
+non-null `admin_base_url`, positive `world_size`, and no `error` before Prime is
+launched.
+
+## Recipes
+
+- [`qwen3_06b_math`](qwen3_06b_math): single-GPU trainer and aggregate Dynamo
+  inference smoke test.
+- [`qwen3_30b_Thinking`](qwen3_30b_Thinking): Qwen3-30B Thinking math with an
+  external prefill/decode deployment.
+- [`glm52_fp8_r2e`](glm52_fp8_r2e): multi-node GLM-5.2 FP8 R2E training with a
+  separately managed DGD.

examples/dynamo/glm52_fp8_r2e/README.md

@@ -0,0 +1,128 @@
+# GLM-5.2 FP8 R2E with external Dynamo inference
+
+This three-step smoke recipe runs a distributed Prime trainer and orchestrator
+against a separately managed Dynamo deployment serving
+`zai-org/GLM-5.2-FP8`. Dynamo owns the frontend, vLLM engines, and one native
+gRPC sidecar per engine group; Prime discovers the mutable engine control
+surface through `/v1/rl/workers`.
+
+The recipe is split into trainer and orchestrator files because the external
+Dynamo DGD and the multi-node trainer have independent lifecycles. It does not
+add a Prime inference configuration or require Prime's launcher to manage the
+DGD.
+
+## Reference topology
+
+| Component | Shape | GPUs |
+|---|---|---:|
+| Dynamo prefill | 2 nodes, DP4 x TP2 x PP1 x EP8 | 8 |
+| Dynamo decode | 2 nodes, DP4 x TP2 x PP1 x EP8 | 8 |
+| Prime trainer | 4 nodes, FSDP16 x CP4 x EP8 | 16 |
+
+The checked-in configuration targets this topology; it is not a claim that
+every cluster can use these parallelism dimensions unchanged. The two
+discovery records must report `prefill` and `backend` components with a
+combined `world_size` of 16. If the DGD topology changes, update
+`weight_broadcast.inference_world_size` in both TOML files to the atomic sum
+returned by the same `/v1/rl/workers` response.
+
+The inference engines must load
+`prime_rl.inference.vllm.worker.nccl.NCCLWeightUpdateWorker`, expose vLLM's
+admin routes, and run the version-matched `vllm-rs` and
+`dynamo-vllm-sidecar` binaries described in [`../README.md`](../README.md).
+For mutable GLM weight reloads, launch every vLLM rank with `--enforce-eager`.
+When serving a snapshot path, set `--served-model-name
+zai-org/GLM-5.2-FP8`. The tested GLM entrypoint also uses the `glm47` tool
+parser, `glm45` reasoning parser, the model chat template, and complementary
+NIXL `kv_producer`/`kv_consumer` roles for prefill/decode.
+The trainer and every inference engine must also use a compatible NCCL
+transport. Apply cluster-specific settings such as `NCCL_IB_DISABLE`,
+`NCCL_SOCKET_IFNAME`, and the NCCL network plugin consistently on both sides;
+do not force Socket on the trainer while allowing inference to select IB.
+The filesystem rollout transport requires the orchestrator and all trainer
+nodes to mount the same read-write shared output root.
+
+## Configure
+
+Initialize the R2E environment submodule and install its workspace package:
+
+```bash
+git submodule update --init -- deps/research-environments
+uv sync --package prime-rl --package r2e-gym-v1
+```
+
+The taskset intentionally uses the current `r2e-gym-v1` default,
+`PrimeIntellect/R2E-Gym-Subset-Verified`, instead of pinning an older dataset
+override in this recipe.
+
+Replace the checked-in service names when the DGD and trainer use different
+DNS names:
+
+- `model.client.base_url`: Dynamo OpenAI frontend on port 8000;
+- `model.client.dynamo_discovery_url`: Dynamo RL discovery on port 8001;
+- `weight_broadcast.host`: trainer rank zero, reachable from every inference
+  engine.
+
+The R2E harness uses Prime sandboxes. Configure the normal Prime credentials,
+or replace the runtime with the sandbox backend used by your cluster. Model
+and dataset caches should be shared across the trainer and inference nodes.
+
+Multi-turn affinity is not enabled merely by sending a session header. Start
+the Dynamo frontend with `--router-session-affinity-ttl-secs <seconds>` (or
+`DYN_ROUTER_SESSION_AFFINITY_TTL_SECS`) and choose an idle TTL longer than the
+longest expected R2E turn gap. Prime maps each trajectory ID to the canonical
+`X-Dynamo-Session-ID` header in `orchestrator.toml`.
+
+Verify both the model and the complete atomic worker snapshot before starting
+Prime. A successful HTTP status alone is insufficient:
+
+```bash
+MODEL=zai-org/GLM-5.2-FP8
+curl -fsS http://dynamo-frontend:8000/v1/models |
+  jq -e --arg model "$MODEL" '.data | any(.id == $model)'
+curl -fsS http://dynamo-frontend:8001/v1/rl/workers |
+  jq -e --arg model "$MODEL" '
+    .protocol_version == 1 and
+    (.workers | length == 2) and
+    (all(.workers[]; .model == $model and
+      ((.error // "") == "") and
+      (.instance_id != null) and
+      ((.admin_base_url // "") != ""))) and
+    ([.workers[].instance_id] | unique | length == 2) and
+    ([.workers[].admin_base_url] | unique | length == 2) and
+    ([.workers[] | select(.model == $model) | .component] | sort == ["backend", "prefill"]) and
+    ([.workers[] | select(.model == $model) | .world_size] | add == 16)
+  '
+```
+
+## Run
+
+Launch the trainer on four 4-GPU nodes with the cluster's distributed runner.
+For example, rank zero's rendezvous address can be passed to `torchrun` while
+all ranks consume the same trainer file:
+
+```bash
+uv run torchrun \
+  --nnodes=4 --nproc-per-node=4 \
+  --rdzv-backend=c10d --rdzv-endpoint="$TRAINER_RANK_ZERO:29501" \
+  --node-rank="$NODE_RANK" \
+  -m prime_rl.trainer.rl.train \
+  @ examples/dynamo/glm52_fp8_r2e/trainer.toml \
+  --output-dir /shared/glm52-dynamo-r2e/train
+```
+
+After trainer rank zero opens port 29500, launch the orchestrator once:
+
+```bash
+uv run orchestrator \
+  @ examples/dynamo/glm52_fp8_r2e/orchestrator.toml \
+  --output-dir /shared/glm52-dynamo-r2e/train/run_0
+```
+
+The gate succeeds when the first optimizer step completes, policy version 1
+settles on all 16 inference ranks through NCCL, and a later multi-turn rollout
+completes without changing its Dynamo session assignment. Three steps are
+required because finite NCCL runs skip broadcasts once
+`step >= max_steps - 1`; this leaves step 1 as the first non-final broadcast
+slot. The disabled post-batch zero-advantage filter keeps this small smoke run
+from stalling on a homogeneous batch; enable it for a production training run.

examples/dynamo/glm52_fp8_r2e/orchestrator.toml

@@ -0,0 +1,60 @@
+max_steps = 3
+batch_size = 2
+group_size = 2
+seq_len = 32768
+max_inflight_episodes = 2
+max_off_policy_steps = 1
+tasks_per_minute = 1
+
+[model]
+name = "zai-org/GLM-5.2-FP8"
+
+[model.client]
+base_url = ["http://dynamo-frontend:8000/v1"]
+dynamo_discovery_url = "http://dynamo-frontend:8001"
+wait_for_ready_timeout = 7200
+
+[model.client.extra_headers_from_state]
+X-Dynamo-Session-ID = "trajectory_id"
+
+[tokenizer]
+name = "zai-org/GLM-5.2-FP8"
+
+[renderer]
+name = "glm-5.1"
+clear_thinking = false
+
+[train.sampling]
+temperature = 1.0
+max_completion_tokens = 2048
+extra_body = { chat_template_kwargs = { clear_thinking = false } }
+
+[[train.source]]
+name = "r2e"
+group_size = 2
+serve.pool = { type = "static", num_workers = 2 }
+env.taskset = { id = "r2e-gym-v1" }
+env.agent.max_turns = 64
+env.agent.max_input_tokens = 30720
+env.agent.max_output_tokens = 16384
+env.agent.max_total_tokens = 32768
+env.agent.timeout = { setup = 600, rollout = 1800, finalize = 300, scoring = 600 }
+env.agent.harness = { id = "bash", edit = true }
+env.agent.runtime = { type = "prime", labels = ["glm52-dynamo"], cpu = 4, creates_per_min = 2 }
+
+[[post_batch_filters]]
+type = "zero_advantage"
+enforce = false
+
+[weight_broadcast]
+type = "nccl"
+host = "trainer-0.trainer-headless"
+port = 29500
+timeout = 12000
+inference_world_size = 16
+
+[rollout_transport]
+type = "filesystem"
+
+[log]
+level = "debug"

examples/dynamo/glm52_fp8_r2e/trainer.toml

@@ -0,0 +1,47 @@
+max_steps = 3
+dist_timeout_seconds = 12000
+
+[model]
+name = "zai-org/GLM-5.2-FP8"
+seq_len = 32768
+impl = "custom"
+attn = "flash_attention_2"
+dp_replicate = 1
+cp = 4
+ep = 8
+optimization_dtype = "bfloat16"
+reduce_dtype = "bfloat16"
+moe_router_dtype = "float32"
+fp8 = false
+optim_cpu_offload = true
+fused_lm_head_token_chunk_size = 1024
+
+[model.ac]
+freq = 1
+
+[model.ac_offloading]
+max_inflight_activations = 1
+
+[tokenizer]
+name = "zai-org/GLM-5.2-FP8"
+
+[optim]
+type = "sign_sgd"
+lr = 1e-6
+weight_decay = 0.0
+
+[scheduler]
+type = "constant"
+
+[weight_broadcast]
+type = "nccl"
+host = "0.0.0.0"
+port = 29500
+timeout = 12000
+inference_world_size = 16
+
+[rollout_transport]
+type = "filesystem"
+
+[log]
+level = "debug"

examples/dynamo/qwen3_06b_math/README.md

@@ -0,0 +1,45 @@
+# Qwen3 0.6B math with external Dynamo inference
+
+This four-step smoke recipe runs the Prime trainer and orchestrator locally while generation is served by an already-running Dynamo frontend, vLLM sidecar, and vLLM engine. Prime does not launch local inference, so the configuration intentionally has no `[inference]` block.
+
+## Prerequisites
+
+Use this version-matched native-gRPC source set:
+
+- Dynamo `feat/dyn-pi-sidecar-v2-review-001` at `836fe81012`
+- vLLM `feat/dyn-pi-sidecar-v2-review-001` at `e56ee21b2c`
+- Prime `feat/dyn-pi-sidecar-v2-review-001`
+
+Build `vllm-rs` and `dynamo-vllm-sidecar` from those revisions into the same
+runtime image. For Kubernetes, start from Dynamo's
+`examples/backends/vllm/deploy/sidecar_agg.yaml`; its adjacent `README.md`
+documents the paired-binary image. Then apply the required Prime RL discovery
+and admin overlay in [`../README.md`](../README.md). This contract requires both
+native gRPC and the Dynamo `/v1/rl/workers` endpoint; a standard Python-only
+vLLM worker is not compatible.
+
+Install the math environment:
+
+```bash
+prime env install primeintellect/math-env
+```
+
+Start an aggregated DP1 Dynamo deployment for `Qwen/Qwen3-0.6B`. The orchestrator waits for both model publication and worker discovery. These requests are useful diagnostics:
+
+```bash
+curl http://127.0.0.1:8000/v1/models
+curl http://127.0.0.1:8001/v1/rl/workers
+```
+
+The checked-in URLs assume Dynamo is reachable from the trainer through localhost, as in a shared dev pod. For a remote DGD, replace both URLs with its frontend services. Also replace `weight_broadcast.host` with a trainer hostname or IP reachable from every sidecar; localhost is not valid across pods or nodes.
+
+`inference_world_size` must equal the sum of `world_size` in one `/v1/rl/workers` response. This recipe assumes one aggregated DP1 engine and therefore uses `1`.
+
+## Run
+
+```bash
+uv run rl @ examples/dynamo/qwen3_06b_math/rl.toml \
+  --output-dir outputs/dynamo-qwen3-06b-math
+```
+
+The run is successful when four optimizer steps complete, the verifier reports math rewards, weight versions advance after each update, and the Dynamo workers remain healthy.