perf: perf recipe changes to enable force on policy ratio

[seonjinn]

What does this PR do ?

This PR updates GB200/H100 GRPO performance recipes for LLaMa-3 8B, Qwen3-32B, Qwen3-30BA3B:

• Enable loss_fn.force_on_policy_ratio: true.
• Set policy.train_global_batch_size: 2048 to align policy training with the generated batch size.
• Cover Llama 3.1 8B, Qwen3-30B-A3B, and Qwen3-32B in both Sync and Async-1off modes.

Across GB200 and H100, enabling force-on-policy ratio:

• reduces logprob time by 41.6%–50.1% and increases logprob throughput by 70.4%–100.5%;
• improves E2E throughput by 7.3%–20.9% in all six GB200 cases and by 5.5%–18.8% in five of six H100 cases;
• leaves policy-training performance largely unchanged, generally within ±3%;

GB200 results

Benchmark setup

• 20 training steps per run
• Reported values: arithmetic mean of steps 3–8 inclusive

The GB200 comparison includes both the GBS increase and force-on-policy ratio; it is not a same-GBS isolation.

E2E impact

Model	Mode	Step time (s)	Delta	E2E tok/s/GPU	Delta
Llama 3.1 8B	Sync	68.202 → 63.307	-7.18%	4,335 → 4,649	+7.25%
Llama 3.1 8B	Async-1off	61.422 → 53.034	-13.66%	5,089 → 5,892	+15.77%
Qwen3-30B-A3B	Sync	196.702 → 178.055	-9.48%	2,115 → 2,340	+10.62%
Qwen3-30B-A3B	Async-1off	271.973 → 231.400	-14.92%	1,531 → 1,800	+17.56%
Qwen3-32B	Sync	310.435 → 277.021	-10.76%	1,360 → 1,525	+12.13%
Qwen3-32B	Async-1off	161.853 → 133.918	-17.26%	1,306 → 1,580	+20.94%

Logprob impact

Model	Mode	Logprob time (s)	Delta	Logprob tok/s/GPU	Delta
Llama 3.1 8B	Sync	11.174 → 6.521	-41.64%	26,435 → 45,042	+70.39%
Llama 3.1 8B	Async-1off	20.768 → 11.524	-44.51%	30,115 → 54,176	+79.89%
Qwen3-30B-A3B	Sync	38.091 → 20.155	-47.09%	10,934 → 20,709	+89.41%
Qwen3-30B-A3B	Async-1off	78.776 → 39.823	-49.45%	10,572 → 20,925	+97.94%
Qwen3-32B	Sync	62.938 → 31.405	-50.10%	6,710 → 13,455	+100.52%
Qwen3-32B	Async-1off	57.005 → 29.693	-47.91%	7,419 → 14,241	+91.95%

Policy-training impact

Model	Mode	Policy time (s)	Delta	Train tok/s/GPU	Delta
Llama 3.1 8B	Sync	19.089 → 18.824	-1.39%	15,545 → 15,652	+0.69%
Llama 3.1 8B	Async-1off	38.395 → 39.236	+2.19%	16,288 → 15,944	-2.11%
Qwen3-30B-A3B	Sync	82.145 → 81.594	-0.67%	5,068 → 5,110	+0.82%
Qwen3-30B-A3B	Async-1off	189.689 → 188.062	-0.86%	4,392 → 4,430	+0.87%
Qwen3-32B	Sync	108.271 → 107.839	-0.40%	3,901 → 3,919	+0.46%
Qwen3-32B	Async-1off	102.177 → 100.766	-1.38%	4,139 → 4,197	+1.40%

H100 results

Benchmark setup

• 20 training steps per run
• Reported values: arithmetic mean of steps 3–8 inclusive

E2E impact

This table isolates the force-on-policy effect by comparing the GBS2048 control against the GBS2048 force run.

Model	Mode	Step time (s)	Delta	E2E tok/s/GPU	Delta
Llama 3.1 8B	Sync	90.856 → 86.836	-4.42%	1,773.4 → 1,870.2	+5.46%
Llama 3.1 8B	Async-1off	64.011 → 53.254	-16.81%	2,577.1 → 3,060.6	+18.76%
Qwen3-30B-A3B	Sync	186.186 → 174.127	-6.48%	1,134.6 → 1,214.3	+7.02%
Qwen3-30B-A3B	Async-1off	142.432 → 142.512	+0.06%	1,504.9 → 1,500.2	-0.31%
Qwen3-32B	Sync	365.451 → 324.271	-11.27%	586.9 → 662.1	+12.82%
Qwen3-32B	Async-1off	193.779 → 164.514	-15.10%	555.4 → 654.8	+17.90%

Logprob impact

Model	Mode	Logprob time (s)	Delta	Logprob tok/s/GPU	Delta
Llama 3.1 8B	Sync	13.204 → 7.547	-42.85%	12,205 → 21,506	+76.21%
Llama 3.1 8B	Async-1off	22.464 → 12.276	-45.35%	14,691 → 26,558	+80.78%
Qwen3-30B-A3B	Sync	25.390 → 13.551	-46.63%	8,321 → 15,604	+87.52%
Qwen3-30B-A3B	Async-1off	44.042 → 22.863	-48.09%	9,622 → 18,518	+92.47%
Qwen3-32B	Sync	77.271 → 38.689	-49.93%	2,777 → 5,553	+99.94%
Qwen3-32B	Async-1off	76.323 → 38.916	-49.01%	2,820 → 5,518	+95.64%

Policy-training impact

Model	Mode	Policy time (s)	Delta	Train tok/s/GPU	Delta
Llama 3.1 8B	Sync	21.011 → 21.564	+2.63%	7,674 → 7,530	-1.87%
Llama 3.1 8B	Async-1off	38.789 → 38.350	-1.13%	8,511 → 8,505	-0.07%
Qwen3-30B-A3B	Sync	39.335 → 39.101	-0.60%	5,373 → 5,407	+0.63%
Qwen3-30B-A3B	Async-1off	76.141 → 76.605	+0.61%	5,563 → 5,526	-0.66%
Qwen3-32B	Sync	115.524 → 112.339	-2.76%	1,858 → 1,912	+2.95%
Qwen3-32B	Async-1off	113.624 → 114.417	+0.70%	1,894 → 1,877	-0.94%

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[seonjinn]

/ok to test e00632b