[BREAKING][algo] feat: Rollout Correction for General Off-Policy Problems#3984

Merged

ISEEKYAN merged 35 commits intoverl-project:mainfrom

szrlee:yingru/rollout_correction

Nov 7, 2025

Commits on Nov 2, 2025

refactor(rollout): migrate to nested config structure without backward compatibility
szrlee
committed

Commits on Nov 6, 2025

fix(config): add required blank line after rollout_correction
szrlee
committed
fix(rollout_corr): prevent silent failure when rollout_rs enabled without threshold
szrlee
committed
feat(rollout_corr): add bypass mode to skip old_log_prob computation
szrlee
committed
refactor(rollout_corr): migrate to typed config with preset methods
szrlee
committed
feat(rollout_corr): add χ² divergence and remove mismatch prefix
szrlee
committed
docs(rollout_corr): add mathematical formulations for general off-policy RL
szrlee
committed
fix(rollout_corr): improve table precision and LaTeX consistency in math doc
szrlee
committed
docs(rollout_corr): fix diagnostic metrics notation
szrlee
committed
docs(rollout_corr): remove bias/variance claims and improve precision
szrlee
committed
docs(rollout_corr_math): update backend example from PyTorch to SGLang
szrlee
committed
fix(config): add _target_ to rollout_correction example, set geo_mis lower threshold, update doc date label
szrlee
committed
docs(rollout_corr): improve math doc - fix notation, add decoupled PPO context
szrlee
committed
docs(rollout_corr): align usage guide terminology with math foundations
szrlee
committed
docs(rollout_corr): clarify decoupled PPO as batch size invariance solution
szrlee
committed
docs(rollout_corr): replace subjective recommendations with objective descriptions
szrlee
committed
docs(rollout_corr_math): fix notation - use prox and C_* constants
szrlee
committed
docs(rollout_corr): fix documentation issues and align with math formulations
szrlee
committed
docs(rollout_corr): fix links, terminology, and clarify PPO is not at fault
szrlee
committed
docs(rollout_corr): align usage guide with mathematical formulations
szrlee
committed
fix(rollout_corr): compute metrics before truncation and improve config usability
szrlee
committed
config(ppo_megatron_trainer): align rollout_correction with ppo_trainer
szrlee
committed
feat(rollout_corr): implement use_pure_rollout_correction parameter
szrlee
committed
docs(rollout_corr): add missing geometric mode in mode combinations table
szrlee
committed
docs(rollout_corr): fix bypass mode example nesting
szrlee
committed
docs(rollout_corr): remove subjective 'Applicable scenarios' claims
szrlee
committed
docs(rollout_corr): remove subjective guidance from metrics documentation
szrlee
committed
feat: move algorithm.rollout_correction to a config snippet
tongyx361
committed
chore: pre-commit
tongyx361
committed
Merge pull request #2 from tongyx361/tyx/feat/off-policy-config-snippet
szrlee
authored
refactor(rollout_corr): remove unnecessary config restoration in pure IS mode
szrlee
committed
fix: rollout_correction
tongyx361
committed
feat: clean ray_trainer
tongyx361
committed

Commits on Nov 7, 2025