fix(ppo): preserve raw KL metric tensor

[EazyReal]

Port of THUDM/slime#2114 for miles' refactored advantage helper. Recut onto current main.

Summary:

• build PPO token-level rewards out of place instead of mutating the kl tensor list
• preserve the raw approximate KL tensor used by rollout/kl logging
• add a CPU test directly on compute_advantages

Validation:

• test fails on the old k *= kl_coef in-place loop
• test passes after the out-of-place multiply

[gemini-code-assist]

Code Review

This pull request prevents the in-place mutation of the input kl list elements in compute_advantages by creating a new token_level_rewards variable instead of modifying the elements in-place. Additionally, a unit test is added to verify that the raw KL metric is preserved. I have no feedback to provide as there are no review comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[EazyReal]

@fzyzcjy @yueming-yuan @maocheng23 @Zhichenzzz @Shi-Dong Could I get a quick review on this raw KL metric fix? It keeps rollout/KL logging raw while training can still use the shaped reward.