[faresobeid]

Engines label each sampled token with the policy step that produced it, and the orchestrator logs per-token policy age.

• /update_weights takes weight_version and calls vLLM's update_weight_version after loading. /load_lora_adapter takes a weight_version query parameter and does the same. The orchestrator sends the step on both (filesystem, NCCL and NIXL transports).
• vLLM 0.32 (on main) then returns weight_versions spans on every /inference/v1/generate choice; renderers/verifiers on main already pass them through to the trace.
• token_policy_ages turns the spans on a trace's assistant nodes into per-token ages against the batch's training version. The orchestrator logs off_policy/token/{mean,max} next to the episode-level off_policy/*.
• The startup sync goes through /update_weights before the dispatcher starts, so the boot step is labelled too. Only Dynamo engines (weight updates via /collective_rpc) keep the "default" label; those tokens are not counted.

Validation

• pytest tests/unit/orchestrator tests/unit/test_configs.py tests/unit/inference -m "not gpu" (445 passed), ruff, uv lock --check.
• GPU (1×H200, Qwen3-0.6B, vLLM 0.32): a 4000-token generation paused mid-way, /update_weights with weight_version=5, resumed. It returned [{"version": "default", "start": 0, "end": 1390}, {"version": "5", "start": 1390, "end": 4000}]; the next request returned [{"version": "5", "start": 0, "end": 8}].

Upstream: vllm-project/vllm#59582 (in 0.32). #4143 is stacked on this PR.

---

Note

Medium Risk
Extends the weight-update and LoRA reload admin APIs to require weight_version and invoke vLLM's update_weight_version; a mismatch or missing field would break sync, but behavior is additive for training.

Overview
Wires vLLM 0.32 weight_versions through the weight-sync path so each sampled token is tagged with the policy step that produced it, and the orchestrator can report finer-grained off-policy signal than episode-level staleness.

After full weight loads (/update_weights) and in-place LoRA reloads (/load_lora_adapter), engines call update_weight_version with the training step the orchestrator already passes on those admin calls (filesystem / NCCL / NIXL and LoRA filesystem receive). Generate responses then carry per-token version spans on traces.

New helpers node_policy_versions and token_policy_ages turn those spans into per-token ages vs the batch’s training policy; at ship time the orchestrator logs off_policy/token/mean and off_policy/token/max alongside existing episode off_policy/* metrics. Tokens still labelled default (e.g. Dynamo engines that skip this path) are excluded from the token metrics.

^{Reviewed by Cursor Bugbot for commit 0025738. Bugbot is set up for automated code reviews on this repo. Configure here.}