[faresobeid]

max_off_policy_steps now bounds the age of each trained token instead of each episode.

On main, the dispatcher cancels every train group whose dispatch version is older than (step-1) - max_off_policy_steps, including finished members, and the sink drops queued traces by the same rule. A group ages out as a whole even when most of its tokens came from recent weights (in-flight keep-mode updates).

With this PR:

• Each sampled token carries the policy step that sampled it (the weight_versions spans from feat: label sampled tokens with the policy step and log per-token policy age (vLLM 0.32 weight_versions) #4141, on main). Unlabelled tokens (Dynamo only) take their episode's dispatch version, which is the old rule for them.
• The sink masks tokens older than (step-1) - max_off_policy_steps out of mask, rl_weights, ce_weights and ref_kl_weights, at the queue sweep and at batch ship. Fresher tokens of the same sample keep training with the sample's advantage and their own inference logprobs.
• A sample is dropped only when nothing is left to train; a trace only when all its samples are dropped (off_policy/dropped).
• Rollouts are never cancelled for staleness. The dispatcher's stale cancel, the stale cancel reason and the sink's whole-group void are removed. On Dynamo, a stale group now runs to completion and is dropped at the sink.
• New metric off_policy/token/masked: fraction of the batch's sampled tokens masked for staleness.
• No loss changes: every loss in trainer/rl/loss.py is token-level and reads the loss mask.

Validation: pytest tests/unit/orchestrator tests/unit/test_configs.py tests/unit/inference tests/unit/eval -m "not gpu" (512 passed), ruff. Not run end-to-end on GPU.

---

Note

Medium Risk
Changes off-policy enforcement in the orchestrator training path (masking vs cancellation), which can alter throughput, batch composition, and train–inference mismatch; loss code is unchanged but what reaches the trainer does.

Overview
max_off_policy_steps now limits staleness per sampled token, not per rollout or group. The train sink masks tokens older than (step-1) - max_off_policy_steps out of mask, rl_weights, ce_weights, and ref_kl_weights (using vLLM weight_versions; unlabeled Dynamo tokens use the episode dispatch version). Fresher tokens in the same sample still train; traces drop only when every sample has nothing left to train.

Rollouts are no longer cancelled for staleness. The dispatcher’s pre–weight-update stale group drops, the stale CancelReason, and whole-group voiding in the sink are removed; on_version_pending only blocks scheduling across the swap. Queue sweeps and batch shipment apply the token masks instead of episode-level drops.

Observability: new off_policy/token/masked rate; docs/config describe token-level behavior; cancellation metrics treat pipeline voids as overload/shutdown rather than stale.

^{Reviewed by Cursor Bugbot for commit 24b8b51. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 3f05442. Configure here.}