[yph22]

What does this PR do?

Add distillation.distillation_loss.loss_mode=resopd, implementing ResOPD: Tail Residualization for Sparse On-Policy Distillation from the implementation. ResOPD integrates the teacher top-k head and aggregate tail analytically and samples only the within-tail residual, estimating the full-vocabulary reverse-KL gradient.

Use an existing synchronous OPD configuration with:

distillation.distillation_loss.loss_mode: resopd
distillation.distillation_loss.topk: 4
distillation.distillation_loss.use_policy_gradient: false
distillation.distillation_loss.use_task_rewards: false
distillation.distillation_loss.loss_max_clamp: null
distillation.distillation_loss.log_prob_min_clamp: null
actor_rollout_ref.model.use_fused_kernels: false
actor_rollout_ref.actor.use_torch_compile: false

See docs/algo/opd.md for the complete configuration overrides.

Duplicate-work check

Searched ResOPD, distillation, and tail residual; inspected the related PRs and #6676 discussion.

• [algo, fsdp, megatron, veomni] feat: add tail-aware top-k KL for on-policy distillation #7855 adds coarse forward KL with a tail bucket; it does not sample the within-tail residual to estimate full reverse KL.
• [trainer] feat: add reverse KL top-k distillation loss #7590 / [algo, trainer, rollout, vllm] feat: add reverse KL top-k with fixed-token prefill scoring #7806 implement truncated reverse KL on the student top-k support. ResOPD uses teacher top-k plus the actual sampled token and does not require the fixed-token scoring extension in [algo, trainer, rollout, vllm] feat: add reverse KL top-k with fixed-token prefill scoring #7806.

Test

Python 3.12.12, PyTorch 2.10.0, CPU only, one thread:

CUDA_VISIBLE_DEVICES= OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 \
  PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest -q \
  tests/trainer/test_resopd_on_cpu.py \
  tests/trainer/test_resopd_integration_on_cpu.py \
  tests/workers/rollout/test_vllm_weight_update_utils_on_cpu.py
# 45 passed

# With the same CPU/thread environment:
python -m pytest tests/workers/test_distillation_topk_symmetry_on_cpu.py -q
# 6 passed

Coverage: independent per-action gradient formula and expected full reverse-KL gradient (all five actions, four support sizes, three seeds), teacher detachment, low precision, negative surrogate values, teacher requests, causal response masking, sequence-parallel slicing/padding, and unchanged legacy parser behavior. Teacher RPC and SP ranks are mocked; no model weights are loaded.

All 14 configured pre-commit hooks passed on the changed files, including generated-config verification, Ruff, mypy, license and compilation checks. git diff --check and the PR-title validator passed. An AST comparison confirms that resopd_loss has identical logic to the authors' release.

An end-to-end GPU training run of this upstream port has not been performed. CPU validation used PyTorch 2.10.0, rather than the current upstream lock's 2.13.0; the repository dependency files are unchanged.

Checklist Before Submitting

• Read CONTRIBUTING.md and AGENTS.md; checked for duplicate work.
• Added documentation and CPU tests discovered by the existing *_on_cpu.py workflow.
• Pre-commit checks on all changed files.
• Reviewed by the ResOPD author; the requested documentation edit is included.
• Maintainer CI authorization, if required.

AI assistance: this port and its tests were prepared with OpenAI Codex assistance at the ResOPD author's request.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: ae75439f-cd4d-4dde-881d-212276e95885

📥 Commits

Reviewing files that changed from the base of the PR and between 770fdff and fef6c71.

📒 Files selected for processing: 8

• docs/algo/opd.md
• tests/workers/rollout/test_vllm_weight_update_utils_on_cpu.py
• verl/trainer/distillation/fsdp/losses.py
• verl/trainer/distillation/losses.py
• verl/trainer/distillation/resopd.py
• verl/workers/config/distillation.py
• verl/workers/rollout/vllm_rollout/utils.py
• verl/workers/rollout/vllm_rollout/vllm_async_server.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Summary

Summary by CodeRabbit

• New Features
  • Added ResOPD distillation with teacher top-k and observed-token scoring for supported training backends.
  • Added optional observed prompt-token scores alongside top-k prompt log probabilities.
• Documentation
  • Added guidance on ResOPD behavior, numerical handling, metrics, and configuration.
• Bug Fixes
  • Improved validation and error reporting for unsupported ResOPD configurations and malformed teacher-score data.
• Tests
  • Added CPU coverage for ResOPD losses, integrations, and prompt-logprob extraction.

This change adds ResOPD loss calculation and connects it to teacher prompt-logprob extraction and supported distillation backends. It adds configuration validation, loss collection, documentation, and CPU tests for estimator behavior and trainer integration.

Changes: ResOPD distillation

Layer / File(s)	Summary
ResOPD loss and configuration verl/trainer/distillation/resopd.py, verl/workers/config/distillation.py, tests/trainer/test_resopd_on_cpu.py, tests/trainer/test_resopd_integration_on_cpu.py, tests/special_sanity/check_license.py	Adds the ResOPD estimator and rejects unsupported configurations. CPU tests cover gradients, numerical behavior, and input validation.
Observed-token score retrieval verl/experimental/teacher_loop/teacher_manager.py, verl/workers/rollout/vllm_rollout/*, tests/workers/rollout/test_vllm_weight_update_utils_on_cpu.py, tests/trainer/test_resopd_integration_on_cpu.py	For ResOPD, requests the observed prompt token’s score from vLLM and appends it after the top-k entries. Tests cover sampling parameters and prompt-logprob extraction.
Trainer integration and validation verl/trainer/distillation/losses.py, verl/trainer/distillation/fsdp/losses.py, tests/trainer/test_resopd_integration_on_cpu.py, docs/algo/opd.md	Routes ResOPD through FSDP, validates teacher payloads, and collects padded losses and diagnostic metrics. Documentation describes the estimator and configuration requirements.

Sequence Diagram(s)

sequenceDiagram
  participant TeacherManager as _get_teacher_sampling_params
  participant VLLMServer as VLLMHttpServer.generate
  participant PromptLogprobs as extract_prompt_logprobs
  TeacherManager->>VLLMServer: sampling parameters with include_prompt_token=True
  VLLMServer->>PromptLogprobs: extract prompt logprobs with observed token enabled
  PromptLogprobs-->>VLLMServer: top-k entries and observed-token score

Loading

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to fef6c

This adds an opt-in ResOPD distillation loss mode. Other loss modes keep their existing behavior. CPU tests cover the estimator and the trainer integration, and no merge-blocking concern was identified.

Security architecture risk: 🔵 Low · up to fef6c

This is an opt-in data-contract change with explicit compatibility checks. No introduced attack path was established, but deployment-level authorization and some failure behavior remain unverified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The inspected effect is on teacher-score responses and the student training batches consuming them. ResOPD does not select a new service destination or grant additional teacher-selection authority. Deployment-wide tenant and network exposure cannot be established from this path.

Trust Boundaries and Controls

• observed — Configured-key teacher routing is preserved from the base. The consumer validates payload structure but continues to trust teacher-supplied IDs and scores; it does not authenticate their semantic correspondence to the training response. That teacher trust predates this PR and is not evidence of a newly introduced attack path.

Resilience and Maintainability Implications

• observed — Teacher requests use fresh request IDs and construct returned score tensors locally. The inspected aborted-output branch returns no prompt-score fields, while the manager requires those fields before returning tensors; this path fails rather than substituting fabricated or cached scores.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 37.93% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 29 functions across 11 files. (1 skipped:…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Title check		The title clearly and concisely identifies the addition of ResOPD sparse on-policy distillation and accurately reflects the main changeset.
Description check		The description directly explains the ResOPD implementation, configuration, testing, documentation, limitations, and relationship to related work.
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 37.93% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 29 functions across 11 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

• Fix all pre-merge checks with AI

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[JacobHelwig]

Hi @yph22 , I think that this PR has substantial overlap with #7855 cc @wuxibin89

[yph22]

Thanks for pointing this out. I’ve updated this PR against the latest main after #7855 was merged.
#7855 adds the coarse-grained forward-KL loss (forward_kl_topk_tail), while this PR implements ResOPD for the full reverse-KL gradient with a sampled within-tail residual. They share the top-k/tail decomposition but use different objectives and teacher payloads, so ResOPD remains a separate opt-in loss mode.