fix(ci): relax grpo-llama3.2-1b-instruct-1n4g-fsdp2tp1.v3 step-time gate to 16s

[yuki-97]

What does this PR do ?

Relaxes the grpo-llama3.2-1b-instruct-1n4g-fsdp2tp1.v3 step-time gate from 14.5 to 16. Test-script only, no runtime change.

What changed

1. mean(data["timing/train/total_step_time"], -6, -1) bound 14.5 -> 16. The recipe has slowed by ~0.9s since late July. On main CI (nvidia/nemo-rl wandb, 61 runs) the gate value, i.e. the mean over steps 495-499, was:

Window	Runs	Gate value
07-20 .. 08-07	20	13.69
09-10 .. 09-20	17	14.18
10-02 .. 10-12	24	14.61

30 of the latest 60 runs exceed 14.5; the max is 15.56.

2. Where the time goes. prepare_for_generation/total accounts for ~0.7s of the slowdown (2.88 -> 3.59); logprob_inference_prep adds ~0.2s. generation, policy_training and the logprob phases are flat. Its two instrumented sub-steps (transfer_and_update_weights, sync_policy_params) did not get slower, so the extra time is in the part that is not timed separately.

3. Likely trigger, not bisected. prepare_for_generation/total is bimodal (~2.9s and ~3.7s). Before 09-28 both modes appear; since then GB200 runs are almost always in the slow mode, starting with the first run at ebbc8fdc8 (build: bump vllm to 0.29.0, #4080), whose parent window also contains #4129. r0.8.0 includes both. GB300 runs are unaffected (3.3s, gate value 14.18), so only GB200 trips the gate.

🤖 Generated with Claude Code

[yuki-97]

/ok to test ea376e3

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.