Add async non-colated PPO + connect to Gym rollout - #3185

Draft

HeyyyyyyG wants to merge 73 commits into

mainfrom

Commits on Jul 1, 2026

feat(swe_bench): add GB200 (hsg) SWE2 generation-scaling launcher and config
RayenTian
committed
cc99a3b
feat(swe_bench): self-contained hsg SWE launcher (baked-gym image + triton MoE)
RayenTian
committed
9519e04
chore(swe_bench): don't source personal env script; document required creds
RayenTian
committed
1392b13
chore(swe_bench): default model/data to SWE1 step_230 + r2e_gym_subset
RayenTian
committed
d5ee3df
chore(swe_bench): revert default data to balanced_language + swe_public_val
RayenTian
committed
a6039a0

Commits on Jul 2, 2026

feat(swe_bench): nano V3.5 SWE scale-gen launcher with gen-benchmark skip-training

RayenTian
and
claude
committed
7195900

Commits on Jul 7, 2026

try to make nano3.5 swe work on main
Jiaqi Zeng
committed
81f5242
fix: declare missing build deps for custom vLLM container builds
Jiaqi Zeng
committed
b66cb73
fix tools/build-custom-vllm.sh
HeyyyyyyG
committed
c88955d
fix tools/build-custom-vllm.sh, nanobind
HeyyyyyyG
committed
afce932
fix: restrict tool.uv.environments to aarch64 only
HeyyyyyyG
committed
c68f226

Commits on Jul 8, 2026

Revert "fix: restrict tool.uv.environments to aarch64 only"
HeyyyyyyG
committed
dec7b08
Revert "fix tools/build-custom-vllm.sh, nanobind"
HeyyyyyyG
committed
64ad3bc
Revert "fix tools/build-custom-vllm.sh"
HeyyyyyyG
committed
dffe6a1
Revert "fix: declare missing build deps for custom vLLM container builds"
HeyyyyyyG
committed
2b199b7
use cudagraph_mode: PIECEWISE in vllm 0.20.0
HeyyyyyyG
committed
dc7e4f5

Commits on Jul 9, 2026

default mtp training on
HeyyyyyyG
committed
3ae4c5f
Merge origin/main into jiaqiz/ppo-dev
HeyyyyyyG
committed
9a474c4
bump gym
HeyyyyyyG
committed
11b1240

Commits on Jul 10, 2026

feat(ppo): add non-colocated generation and train/inference mismatch diagnostics
HeyyyyyyG
committed
cbb4265
feat: add mask_sample

sdevare-nv
authored and
HeyyyyyyG
committed
04011ac

Commits on Jul 11, 2026

Commits on Jul 12, 2026

Merge branch 'jiaqiz/ppo-dev' of github.com:NVIDIA-NeMo/RL into jiaqiz/ppo-dev
HeyyyyyyG
committed
102da6c

Commits on Jul 13, 2026

Commits on Jul 14, 2026

bump gym
HeyyyyyyG
committed
f70c08c
fix survivorship bias for grpo and add dapo grpo baseline
HeyyyyyyG
committed
3ee78be

Commits on Jul 16, 2026

Commits on Jul 20, 2026

Commits on Jul 31, 2026

chore: ignore local Claude Code state and test scratch artifacts

HeyyyyyyG
and
claude
committed
9be0f8f
feat(critic): offline rollout collection for the decoupled critic pipeline

HeyyyyyyG
and
claude
committed
1a6c298
feat(critic): offline critic pretraining on collected rollout shards

HeyyyyyyG
and
claude
committed
47f8a99
feat(ppo): support the privileged critic in async PPO

HeyyyyyyG
and
claude
committed
4fb64f8
feat(critic): calibration diagnostics and a fail-loud privilege precondition

HeyyyyyyG
and
claude
committed
f750f02
fix: crashes and false preconditions hit in the SWE / 1B runs

HeyyyyyyG
and
claude
committed
2519a03
feat(logging): log mtp_metrics from both PPO loops

HeyyyyyyG
and
claude
committed
b1912be
config: SWE PPO run config + long-sequence memory headroom for 1B megatron

HeyyyyyyG
and
claude
committed
4be6d57
config: run-config tuning for the ultra and 1B math sweeps

HeyyyyyyG
and
claude
committed
380346b
chore: bump Gym submodule (rollout /run retry + sandbox DNS fix)

HeyyyyyyG
and
claude
committed
ac59649

Commits on Aug 5, 2026

ppo: let the critic train for more epochs than the actor

HeyyyyyyG
and
claude
committed
1fa70a8
examples: add PPO nano v3.5 SWE config for the hsg cluster

HeyyyyyyG
and
claude
committed
d8ec189
data: import decord lazily in load_media_from_message

HeyyyyyyG
and
claude
committed
eb028ce
ppo: load critic weights only when resuming from a warm-start seed

HeyyyyyyG
and
claude
committed
4c1626b
examples: cap SWE test execution separately from agent time (cmh)

HeyyyyyyG
and
claude
committed
0003f1b
ppo: report critic/explained_var from the pre-update values

HeyyyyyyG
and
claude
committed
e7ac37b

Commits on Aug 7, 2026

Log pre-whitening advantage statistics from GAE

HeyyyyyyG
and
claude
committed
afc2154
Record pre-whitening advantage scale in the PPO rollout dump

HeyyyyyyG
and
claude
committed
b5eb09e

Commits on Aug 8, 2026

feat(ppo): turn-level critic and turn-level GAE for agentic RL

HeyyyyyyG
and
claude
committed
2f5ff50

Commits on Aug 11, 2026

behaviour kl loss between pi_gen and pi_curr
HeyyyyyyG
committed
76c49ed

Commits on Aug 13, 2026

Commits on Aug 28, 2026

fix privilege critic for r2egym
HeyyyyyyG
committed
7d85a64

Commits on Aug 30, 2026

feat(precision): fp32 LM head on both engines (opt-in), ported from super-v3.5

HeyyyyyyG
and
claude
committed
f338853

Commits on Sep 10, 2026

fix(vllm-http): return 400 for engine-side context-length ValueError instead of opaque 500

HeyyyyyyG
and
claude
committed
23229cd