feat(osworld): OSWorld v2 training stack by jinglinglingling · Pull Request #4581 · NVIDIA-NeMo/RL

jinglinglingling · 2026-10-08T01:27:23Z

Summary

add the logical-owner SingleController/context-compaction foundation and portable OSWorld v2 launcher
add FlashREINFORCE binary-KL gating at threshold 0.01 with prompt-mean-token-mean aggregation
add Dr.GRPO and preserved REINFORCE group-baseline recipe alternatives over the complete logical-owner population
add the offline DAPO-band freezer and RFC0040 iter-881 B8/N8 plus target B17/N16 recipes with R3/router replay and LR 5e-6
propagate active logical generation indices so AgentEnv can initialize one OSWorld task and fork exactly the selected owners
select AgentEnv/OpenSandbox at runtime, mount the pinned CA securely, and preflight the chosen backend

Semantics

B17/N16 gives 272 logical training owners per optimizer step (17 * 16 > 16 * 16)
AgentEnv fork oversampling reuses one initialized task state across configured N owners; it does not add optimizer rows
reinforce_baseline and dr_grpo are separate recipe alternatives

Verification

focused FlashREINFORCE/advantage/CC/dataset suite: 177 passed
AgentEnv recipe composition and rollout metadata suite: 14 passed
qualified-container node preflight and resolved B8/N8 recipe validation passed
AgentEnv submit dry-run passed
live Cell-3 provider fork and real OSWorld task setup/fork smoke passed with zero residual sandboxes
copyright, secret, submodule fast-forward, title, and formatting checks pass on the fork PR

Dependency

Depends on NVIDIA-NeMo/Gym#4209. The Gym submodule points at that PR's tested head commit and can advance to its eventual merge commit during review.

Stack

This PR contains the delivered OSWorld v2 context-compaction foundation starting from official NeMo RL commit 5d49fbf4e74bdd3a6627c48759c4a66b8920743b, followed by the FlashREINFORCE and AgentEnv integration.

copy-pr-bot · 2026-10-08T01:27:27Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

…kkeeping Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

Preserve pending call custody, mask failed logical owners, retain healthy siblings, and drop unusable groups through the existing policy. Keep CC integrity failures fatal and avoid unsafe redispatch of request-local selections. Validation: 251 focused RL tests; seven private live vLLM/native-TQ failure, cancellation, and timeout cases. No checkpoint PR or private validation machinery included. Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

Adapt the continuation fallback and bounded manifest drain proposed in NVIDIA-NeMo#4431. Preserve pending IDs when capture fails or times out: a failure record is not proof that a worker has stopped writing. Ordinary receipt mode masks unresolved capture; CC keeps its existing masking. Add regression coverage for public postprocessing, cancellation, pending write custody, and the real RL root decision through Gym HTTP admission. Document the paired buffered Responses SSE behavior. Co-authored-by: alexandery <alexandery@nvidia.com> Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

Allow omitted HTTP headers and stop requiring the old hinted intermediate candidate. Retain exact committed ancestry, segment counts, accepted-token coverage, rejected-token exclusion, and cleanup assertions. Validation: 113 focused local tests and two GPU smoke runs passed; 124 checkpoint compatibility tests passed with the updated Gym client. Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

Signed-off-by: Linglin Jing <linglinj@oci-nrt-cs-001-vscode-01.cm.cluster> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

jinglinglingling · 2026-10-14T15:22:22Z

Rebase/validation update:

Rebased onto current main (f7f14b5396).
Updated Gym pin to the CI-clean rebased fix(automodel): unbreak L1 by adding the missing dpo model_save_format #4209 head (65c0560412).
Added exact processor-owned media-token masking and pinned the companion MBridge fix in fix(nemotron-omni): gate sound merge on audio inputs Megatron-Bridge#6458.
4-node / 32-GPU OSWorld smoke Job 20192123 completed successfully (ExitCode=0:0): 8/8 rollout groups, 404 valid sequences, 1,098,500 valid tokens, old-policy logprobs, advantage, optimizer step, refit, and step_1 checkpoint all completed.
This sampled batch had all-zero rewards, so expected loss/grad norm were both 0.0; the full training control/data path nevertheless completed.
Checkpoint: /lustre/fs1/portfolios/coreai/projects/coreai_dlalgo_nemorl/users/linglinj/osworld-v2-integration/live-execution/results/osworld-v2-latest-mediamask-20261014T141414Z/checkpoints/step_1.

github-actions · 2026-10-14T15:23:25Z

✅ Submodule Fast-Forward Check Results

Check based on commit: fb8f729 (PR #4581 from linglinj/flashreinforce-agentenv-osworld)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

github-actions · 2026-10-14T15:32:31Z

✅ Submodule Fast-Forward Check Results

Check based on commit: 2d8584a (PR #4581 from linglinj/flashreinforce-agentenv-osworld)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

github-actions · 2026-10-14T15:34:20Z

✅ Submodule Fast-Forward Check Results

Check based on commit: c499f19 (PR #4581 from linglinj/flashreinforce-agentenv-osworld)

✅ Submodules that are properly updated:

Gym: ✅ PR branch is ahead of main branch (fast-forward)
Megatron-Bridge: ✅ PR branch is ahead of main branch (fast-forward)

All submodule changes look good! ✨

jinglinglingling mentioned this pull request Oct 8, 2026

feat(osworld): add FlashREINFORCE AgentEnv recipes jinglinglingling/RL#4

Closed

jinglinglingling force-pushed the linglinj/flashreinforce-agentenv-osworld branch from ff3bffa to e970318 Compare October 8, 2026 01:49

jinglinglingling changed the title ~~feat(osworld): add FlashREINFORCE AgentEnv training~~ feat(osworld): add OSWorld v2 training stack Oct 8, 2026

jinglinglingling changed the title ~~feat(osworld): add OSWorld v2 training stack~~ feat(osworld): OSWorld v2 training stack Oct 8, 2026

aroshanghias-nvd and others added 25 commits October 14, 2026 00:45

feat(cc): reconstruct training segments from ordinary rollout history

c999edd

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

refactor(cc): pin lean Gym client and preserve independent test oracles

e701a35

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

build(cc): pin Gym after current-main conflict resolution

6ba40c7

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

build(cc): pin validated Gym worker-completion fixture

f190992

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

ci: restore Docker-compatible lock metadata and satisfy PR checks

c85d237

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

build(cc): leave Gym and lockfile updates to main

fb62b04

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

refactor(cc): preserve main baseline semantics and trim redundant boo…

383de4e

…kkeeping Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

test(cc): distinguish media integrity guards and update renderer fixture

f5cd676

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(cc): reject unsupported generated history items

ebd36fc

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

feat(cc): reconstruct selected forests with shared loss ownership

a4d5640

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(cc): preserve logical advantages in the extracted stage

ea89a73

Signed-off-by: Ali Roshan Ghias <aroshanghias@nvidia.com> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

feat(osworld): add v2 SingleController training recipe

06e99fe

Signed-off-by: Linglin Jing <linglinj@oci-nrt-cs-001-vscode-01.cm.cluster> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

docs(osworld): add qualified container downloader

9faf3f3

Signed-off-by: Linglin Jing <linglinj@oci-nrt-cs-001-vscode-01.cm.cluster> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

feat(osworld): add portable evaluation launcher

ad07b0a

Signed-off-by: Linglin Jing <linglinj@oci-nrt-cs-001-vscode-01.cm.cluster> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

feat(osworld): add FlashREINFORCE AgentEnv recipes

0740335

Signed-off-by: Linglin Jing <linglinj@oci-nrt-cs-001-vscode-01.cm.cluster> Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(osworld): migrate v2 recipes to latest Gym

7bf1741

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(osworld): discover managed Python in nightly images

cdd28ab

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(osworld): run Ray workers from mounted repo

10670d9

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

test(osworld): preflight custom reasoning parser

4cb004e

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(osworld): align rebased Gym rollout runtime

a493ee1

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(osworld): pin external capture join

177b0a5

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(osworld): bound qualification microbatches

edf3130

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

jinglinglingling added 7 commits October 14, 2026 01:22

fix(osworld): pin latest rebased Gym

7e3235b

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(data-plane): bound packed tensor broadcast staging

f4d884f

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(nemo-gym): restore token-free evaluation fallback

a32d709

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(cc): validate optimizer generation windows

525f113

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(osworld): require initialized Bridge submodules

89518b4

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(osworld): preserve generated media-token IDs as text

aeb7443

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

fix(osworld): align policy sampling temperature

fb8f729

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

jinglinglingling force-pushed the linglinj/flashreinforce-agentenv-osworld branch from e970318 to fb8f729 Compare October 14, 2026 15:20

github-actions Bot added the Documentation Improvements or additions to documentation label Oct 14, 2026

jinglinglingling mentioned this pull request Oct 14, 2026

feat(osworld): OSWorld environment stack NVIDIA-NeMo/Gym#4209

Open

test(osworld): add required copyright headers

2d8584a

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

chore(osworld): pin CI-clean rebased Gym

c499f19

Signed-off-by: Linglin Jing <linglinj@nvidia.com>

Navigation Menu

Repository navigation

feat(osworld): OSWorld v2 training stack - #4581

feat(osworld): OSWorld v2 training stack#4581
jinglinglingling wants to merge 34 commits into
NVIDIA-NeMo:mainfrom
jinglinglingling:linglinj/flashreinforce-agentenv-osworld

jinglinglingling commented Oct 8, 2026

Uh oh!

copy-pr-bot Bot commented Oct 8, 2026

Uh oh!

jinglinglingling commented Oct 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

github-actions Bot commented Oct 14, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jinglinglingling commented Oct 8, 2026

Summary

Semantics

Verification

Dependency

Stack

Uh oh!

copy-pr-bot Bot commented Oct 8, 2026

Uh oh!

jinglinglingling commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 14, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Oct 14, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

github-actions Bot commented Oct 14, 2026

✅ Submodule Fast-Forward Check Results

✅ Submodules that are properly updated:

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jinglinglingling commented Oct 14, 2026 •

edited

Loading