fix(train): hand off to the engine before an off-cadence final eval

[guapisolo]

Summary

Runs the last rollout's engine handoff whenever a final eval follows, including off-cadence.

Symptom & Reproduction

• Symptom: the 2026-09-22 nightly on main (366f614b8b, run 35745409894, attempt 1) failed test_qwen3_vl_4B_fsdp in stage-c-2-gpu-h200 (0) and test_qwen3_4B_fsdp_true_on_policy in stage-c-2-gpu-h200 (1) with ValueError: Pointer argument cannot be accessed from Triton (cpu tensor?) in write_req_to_token_pool_triton at Eval ... 0%.
• Log evidence: after rollout 2 both logs show release_memory_occupation but no resume_memory_occupation or update_weights_from_tensor before the eval.
• Last pass: both tests passed in ci(docker): bump Megatron-Bridge to bridge-new (CI probe) and testing #3291's CI on 09-21, before Skip the last rollout's engine handoff unless a final eval follows #3182 merged.
• Reproduction: tests/fast/test_train.py::TestFinalEval::test_an_off_cadence_final_eval_follows_the_last_handoff (num_rollout=3, eval_interval=2) fails on main: no onload_weights / update_weights:2 precede eval:2.

Root Cause

1. train.py handoff guard calls should_run_periodic_action without args.num_rollout.
2. should_run_periodic_action forces the last rollout only when num_rollout is passed.
3. At --num-rollout 3 --eval-interval 20, the guard is false on rollout 2; the handoff is skipped.
4. The eval gate passes args.num_rollout, so the final eval still dispatches.
5. release_memory_occupation leaves the colocated engine released; eval requests crash its scheduler.

Fix

train() computes eval_due once and uses it for both the handoff guard and the eval gate, so the two cannot disagree. The last rollout still skips the handoff when no final eval follows, which is #3182's intent. Every earlier rollout is unchanged. Without colocation the old code did not crash, but its final eval scored the previous rollout's weights; that also changes.

Verification

• tests/fast/test_train.py, tests/fast/test_train_async.py: 16 passed locally (CPU). The new off-cadence test fails on main (8cdf3794d9) and passes here; test_the_last_rollout_skips_the_handoff_without_an_eval passes on both.
• The two GPU e2e tests were not rerun; they need run-ci-fsdp.

Review Focus

• eval_due in train.py: the handoff must run for every rollout whose eval runs, and only then on the last one.

🤖 Generated with Claude Code

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[guapisolo]

@claude review always

[guapisolo]

/rerun-test tests/e2e/fsdp/test_qwen3_vl_4B_fsdp.py

[guapisolo]

/rerun-test tests/e2e/fsdp/test_qwen3_4B_fsdp_true_on_policy.py

[claude]

Code review found no issues

No high-confidence issues detected in this change.

[github-actions]

✅ tests/e2e/fsdp/test_qwen3_4B_fsdp_true_on_policy.py passed on stage-c-2-gpu-h200 in 17m02s — workflow run

[github-actions]

✅ tests/e2e/fsdp/test_qwen3_vl_4B_fsdp.py passed on stage-c-2-gpu-h200 in 17m31s — workflow run