[faresobeid]

What

One definition of rollout age, and staleness enforced per episode instead of per group.

rollout_age(start_version, step) = (step - 1) - start_version (orchestrator/utils.py, replaces min_fresh_version): the version batch step trains on minus the version the episode was dispatched at. Queue time counts. The dispatcher early cancel, the sink's insertion check and queue sweep, off_policy/* and dispatcher/off_policy/* all read it.

Each episode is stamped with its own dispatch version. Before, every member got group.policy_version_at_start, even members dispatched several versions later (large G with a binding max_inflight). policy_version_at_start now only sets the group's shared prefix-cache salt.

Per-episode enforcement

• Dispatcher: on_version_pending calls drop_group(gid, reason="stale", stale_only=True) for each group with a stale in-flight episode. That cancels only the group's in-flight episodes past the bound; the rest of the group keeps running. drop_group re-reads inflight at call time, so an episode that completes while an earlier group's cancellation is being emitted is never double-claimed. Without stale_only it cancels everything the group still owes, as before.
• Sink: a stale cancellation no longer voids the group's arrived episodes. The sink keeps several cancellation markers per group and adds up their counts. When the group completes, arrived episodes that are stale themselves are dropped before finalize_group and the curriculum (counted as cancelled, so they show as stale in the discard warning); the rest is scored as usual.

Rule for partial groups

A group with dropped stale members is handled like a group with errored or overload-cancelled members today: it is scored from the fresh members that arrived. GRPO with one fresh member gets advantage 0, so zero-advantage pruning removes it. Stale drops remove the slowest members, so the GRPO baseline is taken over a duration-truncated sample (errors and overload cuts already do this). Before, the stale case avoided that by throwing away the whole group, wasting G-1 finished rollouts. This only changes behavior when members of a group start at different versions.

Also

• max_off_policy_steps docstring, docs/training.md, and the GroupCancellation docstring describe per-episode dropping.

Checks

• pytest -m "not gpu" tests/unit/orchestrator tests/unit/test_configs.py: 291 passed. ruff check / ruff format --check: clean.
• Off-tree sink equivalence (not committed): this head and the previous head of this PR give identical batches, curriculum calls and drop counts on 300 random event sequences (staggered member starts, failures, stale/overload cancels).
• Off-tree dispatcher check (not committed): two groups each with a stale and a fresh member in flight; while the first group's cancellation is emitted, the other group's stale member completes. The previous head raised KeyError in cancel_episodes; this head emits one marker, keeps permits and group accounting consistent, and a later full drop_group frees the group.
• No new tests: the changed logic is dispatcher/sink glue.

---

Note

Medium Risk
Changes off-policy bounds and partial-group scoring in the orchestrator train path; mis-tuning or edge cases around mixed-version groups could alter GRPO baselines and batch composition.

Overview
Off-policy staleness is now defined and enforced per episode, not per rollout group.

A shared rollout_age(start_version, step) replaces the old group-level min_fresh_version cutoff. Each episode is stamped with its own dispatch policy version (policy.version at schedule time); the group’s policy_version_at_start is only used for prefix-cache salting.

Dispatcher: before weight updates, only in-flight train episodes past max_off_policy_steps are cancelled via new cancel_episodes; younger siblings and undispatched group members keep running. Full-group abandonment still goes through drop_group.

Train sink: queue sweeps and group finalization drop individual stale episodes; stale GroupCancellation markers no longer discard the whole group. Multiple cancellation markers per group are summed. Surviving fresh members are scored like partial groups after errors (e.g. GRPO over remaining rollouts).

Docs/config and /pause mode="keep" comments are updated to match freeze-and-resume behavior (not drain).

^{Reviewed by Cursor Bugbot for commit 53dcd07. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 53dcd07. Configure here.}