[algo] SAPO algo by Qwen

[BounharAbdelaziz]

What does this PR do?

Implements Soft Adaptive Policy Optimization (SAPO) for RL fine-tuning of LLMs. SAPO replaces hard clipping with temperature-controlled soft gating for more stable training and better sample efficiency. Paper: https://arxiv.org/abs/2511.20347

Checklist Before Starting

• Search for similar PRs: SAPO search
• Format PR title: [algo] feat: implement SAPO (Soft Adaptive Policy Optimization)

Design & Code Changes

• Added compute_policy_loss_sapo() in verl/trainer/ppo/core_algos.py
• Implements soft gate: f(r) = σ(τ(r-1)) · 4/τ where r = π_θ / π_θ_old
• Uses asymmetric temperatures (τ_neg > τ_pos) as in original paper
• Aggregation: seq-mean-token-mean as per paper

Checklist Before Submitting

• Read Contribute Guide
• Apply pre-commit checks
• Add/update documentation
• Add unit tests (if feasible)
• Request CI in Slack

XP: SAPO vs GRPO

• Setup: Qwen3-4B-Base, total batch size 256, mini-batch size 32, context len 8192.
• Stability: SAPO training is more stable than GRPO.
• Response length: SAPO produces substantially longer responses on average.
• Entropy / collapse: GRPO collapses quickly and its performance saturates early, while SAPO maintains healthier exploration (seen entropy).
• Gradient norm: SAPO runs with slightly higher, but well-behaved, gradient norms.
• PG loss: The PG loss under SAPO stays around −0.01, i.e., the underlying SAPO objective (before the leading minus sign) is positive, indicating the model consistently discovers and reinforces high-reward behaviors.

[gemini-code-assist]

Code Review

This pull request introduces the Soft Adaptive Policy Optimization (SAPO) algorithm by adding a new policy loss function and its configuration. The implementation is mostly correct, but I've found a few critical issues that need to be addressed. Firstly, the new configuration parameters tau_pos and tau_neg are missing from the ActorConfig dataclass, which will cause a crash on startup. Secondly, the loss_agg_mode in the new SAPO loss function is hardcoded, making it non-configurable. Lastly, there's a potential for a division-by-zero error in the gating function that could lead to training failure. Please address these points to ensure the stability and correctness of the new algorithm.

[gemini-code-assist]

The new configuration parameters tau_pos and tau_neg are not defined in the ActorConfig dataclass located in verl/workers/config/actor.py. This will cause a ValidationError when Hydra/OmegaConf attempts to instantiate the ActorConfig from this YAML file, as these are unrecognized fields. To fix this, you need to add these fields to the ActorConfig dataclass definition.

[gemini-code-assist]

The loss_agg_mode is hardcoded as "seq-mean-token-mean" in the call to agg_loss. This completely ignores the loss_agg_mode parameter passed to the compute_policy_loss_sapo function, making this aspect of the loss calculation non-configurable. The function should use the value from the loss_agg_mode argument to allow for flexibility.

Suggested change

	loss_mat=pg_losses, loss_mask=response_mask, loss_agg_mode="seq-mean-token-mean", **config.global_batch_info
	loss_mat=pg_losses, loss_mask=response_mask, loss_agg_mode=loss_agg_mode, **config.global_batch_info

[BounharAbdelaziz]

@vermouth1992 @eric-haibin-lin

[vermouth1992]

Could you please add an example script in the example folder? Thanks

[vermouth1992]

Also, if possible, show a convergence curve of some task

[BounharAbdelaziz]

Could you please add an example script in the example folder? Thanks

@vermouth1992 done!

[BounharAbdelaziz]

Also, if possible, show a convergence curve of some task

@vermouth1992 two ongoing xps right now on dapo comparing vanilla GRPO and SAPO (I'll probably also add one run with GSPO).

[vermouth1992]

Could you please fix sanity and precommit ci? Thanks.

[BounharAbdelaziz]

Could you please fix sanity and precommit ci? Thanks.

@vermouth1992 done!
issue was with the new tau params that needed to be added.
sorry for the delay.

[BounharAbdelaziz]

@vermouth1992 @eric-haibin-lin any update?

[bhaktatejas922]

will this work with VL?

[vermouth1992]

It seems that there are two SAPO recipes. Shall we merge them into one and acknowledge both contribution? #4624

[BounharAbdelaziz]

It seems that there are two SAPO recipes. Shall we merge them into one and acknowledge both contribution? #4624

@CedricHwong could you please clarify the concrete differences you’re seeing between the two? I’m confident we can align them if needed, and I’m happy to give you access so we can work on this together.

Also, note that by pushing my latest commit, some of your previous changes may have been broken making the two PRs the same... and so the behavior you’re referring to might not reflect the current state of the code.

Finally, did you run any tests or training runs with your version that you could share? That would help ground the discussion.

Many thanks!

[BounharAbdelaziz]

@vermouth1992 anything missing to merge this PR? Thanks :)