Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/inference.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,7 @@ On the CLI the same keys are available as `--inference.vllm.max-num-seqs 256` (o

Router replay works by capturing the expert routing decisions into a buffer. This buffer then gets sent to the trainer, which can use it instead of re-computing the routing. This lowers the trainer↔inference mismatch by an order of magnitude, resulting in more stable training.

To enable router replay, you can set `inference.vllm.enable_return_routed_experts = true`. Capture runs on vLLM's V2 model runner (forced automatically). Router replay is not supported with disaggregated P/D: vLLM does not capture routed experts across NIXL KV transfer.
To enable router replay, you can set `inference.vllm.enable_return_routed_experts = true`. Capture runs on vLLM's V2 model runner (forced automatically). Under disaggregated P/D the router takes the prompt routing from the prefill instance and the completion routing from the decode instance. This needs router v0.2.3 or newer: decode responses carry routing rows only from their first forward, and the router cuts them by position.

```toml
[trainer]
Expand All @@ -299,7 +299,7 @@ enable_return_routed_experts = true

This however is not free, it adds a significant overhead to the HTTP requests as this payload can grow quite large. We reccomend sizing up the env server pool (`orchestrator.*.source.serve.pool`) to allow for more parallelization on the verifiers side.

Currently this feature is also not supported with CPU KV cache offload, which can have negative impact on the inference throughput.
Router replay works with `inference.kv_cache_offload` (native and Mooncake). vLLM keeps routing rows in a local store keyed by block hash, and no connector moves them with the KV, so a loaded block has no rows on this instance. prime-rl therefore limits an offload load to the prompt tokens the client already has routing for: the blocks before `routed_experts_prompt_start`. Multi-turn rollouts keep their offload hits, because the reloaded earlier turns sit before that position. Offloaded prefixes past it (first turns, prompts shared across a group or across nodes) are recomputed. Loaded blocks then serve local GPU prefix hits like any other block, so a local hit is cut at the first loaded block that a request needs routing from (for example a first turn that hits a prefix reloaded by another rollout's later turn), and the rest is recomputed. Under P/D, decode-side loads are not limited: decode requests emit routing only from their first forward.

### Sampling Replay

Expand All @@ -313,6 +313,6 @@ top_k = 512 # optional, defaults to 512 under truncation (bounds each sampling

That's all — there are no replay flags. Truncated train sampling makes the inference server return sampling masks (`inference.enable_return_sampling_mask`, auto-enabled), and the trainer replays them. Train-sampling `top_k` above 512 is rejected: the trainer pads each micro batch to its largest sampling mask, so the bound caps trainer memory. Configs that would break under renormalized logprobs are rejected: `opd`/`opsd` and truncation knobs smuggled via `extra_body`. Frozen-source envs are exempt.

Capture runs on vLLM's V2 model runner (forced automatically) and is engine-wide. It can run together with router replay. While sampling capture is on, vLLM rejects any request with `temperature <= 0` or without an effective `top_k > 0` — eval sampling against the training server must set `top_k` (the model's generation config often supplies one) and a non-zero temperature.
Capture runs on vLLM's V2 model runner (forced automatically) and is engine-wide. It can run together with router replay, including under disaggregated P/D. While sampling capture is on, vLLM rejects any request with `temperature <= 0` or without an effective `top_k > 0` — eval sampling against the training server must set `top_k` (the model's generation config often supplies one) and a non-zero temperature.

When launching the inference server standalone, set `inference.enable_return_sampling_mask = true` yourself; clients must sample with `0 < top_k <= 512`.
10 changes: 0 additions & 10 deletions packages/prime-rl-configs/src/prime_rl/configs/inference.py
Original file line number Diff line number Diff line change
Expand Up @@ -508,16 +508,6 @@ def validate_llmd_no_routed_experts(self):
)
return self

@model_validator(mode="after")
def validate_disaggregated_no_routed_experts(self):
"""vLLM rejects routed-expert capture with NIXL KV transfer."""
if self.deployment.type == "disaggregated" and self.vllm.enable_return_routed_experts:
raise ValueError(
"Routed-expert return (enable_return_routed_experts) is not supported with disaggregated P/D: "
"vLLM does not capture routed experts across NIXL KV transfer."
)
return self

@model_validator(mode="after")
def validate_router_deployment(self):
"""The llm-d router (EPP + Envoy) is launched by the SLURM templates only; multi-node deployments need a router to front the per-rank engines."""
Expand Down
28 changes: 0 additions & 28 deletions packages/prime-rl-configs/src/prime_rl/configs/rl.py
Original file line number Diff line number Diff line change
Expand Up @@ -594,34 +594,6 @@ def auto_setup_sampling_mask_capture(self):
)
return self

@model_validator(mode="after")
def validate_disaggregated_no_routed_experts(self):
"""Runs after ``auto_setup_router_replay``, which sets the inference flag after InferenceConfig's validators."""
inference = self.inference
if (
inference is not None
and inference.deployment.type == "disaggregated"
and inference.vllm.enable_return_routed_experts
):
raise ValueError(
"Router replay (inference.vllm.enable_return_routed_experts / trainer.enable_router_replay) "
"is not supported with disaggregated P/D: vLLM does not capture routed experts across NIXL KV transfer."
)
return self

@model_validator(mode="after")
def validate_router_replay_without_kv_offload(self):
if (
self.trainer.enable_router_replay
and self.inference is not None
and self.inference.kv_cache_offload is not None
):
raise ValueError(
"Router replay with inference.kv_cache_offload is not supported. "
"External KV cache hits do not carry routed-expert decisions."
)
return self

@model_validator(mode="after")
def validate_mooncake_offload_requires_slurm(self):
if (
Expand Down
Loading
Loading