feat(rl): add score-centered policy gradients

[nanjiangwill]

Summary

Implement score-centered policy gradients from arXiv:2609.20807 for FSDP and Megatron training.

At every generated-token prefix, the loss subtracts the sampler expectation of the trainer score. Sampler probabilities and optional importance weights are stop-gradient coefficients; gradients flow only through trainer log probabilities.

Dependencies

The branch includes bounded sampling-support replay. Exact-support score centering also uses sgl-project/sglang#40932, which returns the complete normalized sampler behavior distribution aligned with realized-support IDs.

Modes

• Modeled tail: with untruncated, unit-temperature rollout sampling, retain the sampler's highest-probability head and model the omitted tail with the trainer distribution. --score-centering-head-size defaults to 128 and does not alter sampling.
• Exact support: with rollout top-p/top-k, use every token in the realized sampling support. SGLang supplies normalized sampler probabilities q; Miles normalizes trainer logits p on the same replayed support.

Modeled-tail mode currently fails closed at non-unit temperature because SGLang's generic top-logprob response does not expose whether a remote server selected pre- or post-temperature semantics. Exact-support mode uses an explicit behavior-probability contract and supports any positive configured temperature.

Correctness boundaries

• Reject unsupported sampler transforms, beam search, incompatible logprob recomputation, and objectives that do not share the required probability space.
• Preserve singleton distributions for tool and environment tokens, giving them exactly zero policy gradient.
• Compose with built-in TIS and IcePop weights using the paper's weighted-score extension.
• Carry typed sampler metadata through direct rollout, sessions, object-store conversion, FSDP, Megatron, tensor parallelism, and context parallelism.
• Avoid full-vocabulary materialization with sparse vocabulary-parallel logprob/autograd primitives.

Validation

• 555 focused replay, score-centering, argument, rollout, session, transport, and sparse-math tests passed.
• 56 FSDP and Megatron model/lifecycle tests passed with Megatron-LM on PYTHONPATH.
• isort, Black, Ruff, documentation JSON validation, and git diff --check pass.

GPU end-to-end score-centering execution remains to be run.