feat: R3 gym notq router replay

[zyzhou5]

What does this PR do ?

Adds no-TQ async Gym path support for R3 router replay.

When policy.router_replay.enabled=true, async Gym rollouts now request routed expert indices from Gym, validate that they are present, and convert them into Nemo-RL message logs. The routed expert indices are sliced across prompt and generation tokens for each Gym model call so the existing R3 training path can consume them.

This also updates the async OpenAI-compatible vLLM endpoint to return routed expert indices in chat response messages when requested.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[zyzhou5]

/ok to test 763fb26

[zyzhou5]

/ok to test c289b97

[zyzhou5]

/ok to test df70c93

[zyzhou5]

/ok to test f594c2d

[zyzhou5]

/ok to test c25a0bf

[yuki-97]

thanks @zyzhou5 , changes LGTM, some comments on test things.

can we add several nightly tests to guard r3 working correctly? we currently seem only cover grpo TQ path in examples/configs/recipes/llm/grpo-qwen3-30ba3b-8n8g-megatron-cp2-r3.yaml, could you add the below?

1. grpo path: #2849
2. async grpo path: related #2908
3. gym path: this PR

also maybe good to rename examples/configs/recipes/llm/grpo-qwen3-30ba3b-8n8g-megatron-cp2-r3.yaml to keep the same name style like the other TQ tests.

[zyzhou5]

/ok to test 25c2f00

[zyzhou5]

/ok to test 6b8df98

[zyzhou5]

/ok to test 43d34b5

[yuki-97]

/ok to test 58b6af1

[yuki-97]

LGTM, thanks @zyzhou5

[qiaochuz-nv]

2. test_grpo_qwen3_30ba3b_megatron_cp2_r3_async_gym_e2e
   PR covered: NVIDIA-NeMo/RL feat: R3 gym notq router replay #2915

   Functions / paths covered:

   • R3 router replay through NeMo-Gym async generation
   • examples/nemo_gym/run_grpo_nemo_gym.py
   • Gym/OpenAI-compatible server round-trip preserving routed_experts
   • async Gym rollout path
   • Megatron Qwen3-30B-A3B MoE policy path
   • CP2 route sharding
   • non-colocated vLLM/Gym generation
   • train-stage RouterReplay assignment/action
   • forward verifier matches_expected=true
   • CP token identity verification
   • offline SWE1 dataset staging from HF cache
   • CI-scaled 3-node / 24-GPU smoke for upstream 16-node recipe

   Status: PASS
   Evidence:

   • run_id: 20260707T093631Z-6495538f
   • SLURM job: 5584814
   • SLURM: COMPLETED|0:0
   • elapsed: 00:09:14
   • nodes: eos[0143,0192,0255]
   • train/loss steps: 2
   • median(train/token_mult_prob_error): 1.015147626399994
   • R3 trace records: 24640
   • train replay assignments: 6144
   • train replay forward: 6144
   • train replay backward: 6144
   • forward verifier: 6144
   • CP token identity rows: 193480