swe_env/anyswe: address @ffrujeri review on #1572 (fixes + gold baseline) by adil-a · Pull Request #1802 · NVIDIA-NeMo/Gym

adil-a · 2026-06-28T00:55:12Z

Addresses @ffrujeri's review on #1572 (all 13 inline comments) plus the reward-profiling baseline. Targets cmunley1/anyswe.

Note: the swe_env README.md / ng_test_all CI-unblock landed separately via #1796 (merged). This PR is the review-comment fixes + docs + gold baseline on top of that.

Review-comment fixes (@ffrujeri)

#	Comment	Fix
1	truncation mask never fires	inner agents (hermes/claude_code) now emit top-level `status="incomplete"` on internal max-turns/context/timeout; anyswe masks `resolved AND incomplete` (verified the status round-trips through `response.json`). openclaw exposes no internal signal → documented.
2	openclaw config not runnable	select the apptainer provider for its `.sif` formatter; drop dead `apptainer_memory_limit_mb`/`skip_eval` fields.
3	docker timeout gaps	configurable default `exec` timeout + bounded `close`/`cp` so a hung in-container command can't block a rollout.
4	reward-profiling evidence	gold-patch baseline in the swe_env README (below).
5	`_setup_params` assumes str `instance_dict`	accept str or dict (mirror `_build_swetask`).
6	r2egym docstring	missing `eval_script` is unmasked reward-0, not an eval-error mask.
7	missing docker binary	wrap `FileNotFoundError` → clear `SandboxCreateError`.
8	unquoted shell interpolation	`shlex.quote` dataset values in `bash -c` (nv_internal/harness/swe_bench_ext).
9	rc 125/126/127 infra-classify	narrow to rc 125 only (126/127 are legit user-command codes).
10	swe_rebench inflates on empty-required	empty-required guard (consistent with `compute_resolved`).
11	unmasked infra failure on docker grade	mask `SandboxCreateError`/image-pull (`error_kind="sandbox"`).
12	per-framework parsers untested	add `test_parsing_frameworks.py` (the 7 reviewer cases).
13	docs discoverability	add a fern SWE-Environment page.

Gold-patch baseline (review #4) — at parity with the nested reference

A full 500-instance SWE-bench Verified gold-patch census on docker resolves 493/500 (patch_exists 500/500, 0 infra errors), matching the apptainer/.sif nested reference 492/500 to within environment noise; empty patch 0/500. (docker and apptainer both use the host-side flat grader here, so they're 1-1; the .sif figure is swebench's nested run_evaluation.)

The census surfaced + fixed two real flat↔nested reconstruction gaps (445 → 486 → 493), verified with 0 regressions:

PYTEST_ADDOPTS=-rA: swebench 4.1.0's eval for some families (sphinx via tox, several sklearn) runs pytest without -rA, so passing tests print only as dots and the host-side parser saw zero passes → ~45 unresolved even for gold (445→486).
drop GIT_CONFIG_GLOBAL=/dev/null: older images' git can't parse /dev/null, so the eval's git checkout + test-patch git apply failed → required tests "absent" (486→493).

The remaining 7 misses are a small symmetric difference with .sif (4 shared genuine env-flaky: astropy-7606/8707/8872, django-10097; 3 docker-only sphinx instance-specific quirks; docker also resolves 4 that .sif misses).

Validation

258 unit tests pass (swe_env + docker provider + anyswe + claude_code/openclaw); ruff clean; fern check clean.
The #1 truncation logic was adversarially verified end-to-end (top-level status survives the container→host round-trip; the mask is live).

🤖 Generated with Claude Code

…ng tests) - docker provider: configurable default exec timeout + bounded close/cp so a hung in-container command can't block a rollout; clear error when docker is missing (wrap FileNotFoundError); classify only rc 125 (not 126/127 user-command codes) as an infra failure. - verify_task: mask SandboxCreateError / image-pull failures (error_kind=sandbox) so infra hiccups on the docker pull-on-demand path don't depress resolve rate. - swe_rebench: empty-required guard (consistent with compute_resolved) so a degenerate row no longer inflates to reward 1.0. - anyswe _setup_params: accept instance_dict as str OR dict (mirror _build_swetask). - shlex.quote dataset values interpolated into bash -c (nv_internal/harness/ swe_bench_ext). - r2egym docstring: missing eval_script is unmasked reward-0, not an eval-error mask. - openclaw config: select the apptainer provider for its .sif formatter; drop dead apptainer_memory_limit_mb/skip_eval fields. - tests: add test_parsing_frameworks.py (jest/maven/cargo/pytest/unknown); docker regression tests (rc125-only, 126/127 user-errors, default timeout, missing-docker); split verify_task into sandbox-masked vs generic-unmasked; swe_rebench empty-required. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

Adds fern/ Agent Server page covering the harness-agnostic SWE environment (anyswe runner + swe_env library: harnesses, sandbox lifecycle, inline verify_task, self_drive) and selecting the docker vs apptainer sandbox provider. fern check clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

… anyswe mask) anyswe's truncation mask (resolved AND status==incomplete) was inert because no inner agent set the top-level status. Now each self-reports: - hermes: status=incomplete when run_conversation reports completed=False (max-iterations). - claude_code: parse_stream_json flags incomplete on a CLI result event that is_error or has a non-"success" subtype; an internal subprocess timeout (which can fire before anyswe's outer agent timeout) emits a synthetic result event so it self-reports incomplete rather than slipping through both guards. - openclaw: documented that it exposes no internal stop-reason signal (status left unset); its subprocess-timeout truncation is masked upstream by agent_timed_out. Verified end-to-end: top-level status survives the response.json round-trip (model_dump_json -> model_validate_json) and anyswe masks resolved+incomplete. Tests: claude_code result-subtype + synthetic-timeout incomplete cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

… under-resolution) The full SWE-bench Verified gold-patch census on docker exposed a host-side parse gap: swebench 4.1.0's per-repo eval command for some families (sphinx via tox, several sklearn) runs pytest without -rA, so passing tests print only as progress dots. The host-side parser (parse_log_pytest_v2) keys off 'PASSED <nodeid>' lines, so it saw zero passes and marked those instances unresolved even for the gold patch (census: ~45 sphinx + 5 sklearn missed -> 445/500 vs the .sif/nested reference 492/500). Forcing PYTEST_ADDOPTS=-rA in the flat eval makes every test's result visible to the parser; non-pytest families (django's runner) ignore it. Verified: sphinx-doc__sphinx-7440 gold went status_map 1->14, resolved False->True. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

Reward-profiling evidence (review #4): the docker gold-patch census resolves 486/500 (after the PYTEST_ADDOPTS=-rA flat-grading fix; 445 before), in line with the apptainer/.sif reference 492/500; the empty patch resolves 0/500. The remaining ~14 are instance-specific (documented astropy/django upstream flaky gold-failures + a few where a single required test does not run in the container), not a systematic grader defect. Includes a docker/apptainer repro snippet. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

copy-pr-bot · 2026-06-28T00:55:16Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-06-28T00:56:20Z

🌿 Preview your docs: https://nvidia-preview-swe-env-review-fixes.docs.buildwithfern.com/nemo/gym

Here are the markdown pages you've updated:

nemo/gym/agent-server/swe-environment

…ld images) The docker gold census's residual misses vs the .sif/nested 492 reference were NOT concurrency or a docker-vs-apptainer difference — they were a flat-grading reconstruction bug. Older instance images' git cannot parse GIT_CONFIG_GLOBAL=/dev/null ('bad config line 1 in file /dev/null'), so the eval script's git checkout + test-patch git apply FAIL, the gold fix's new tests are never added, and the required tests come back 'absent' -> false unresolved. swebench's nested eval (which produced 492) doesn't null the config. Dropping it (keeping GIT_PAGER=cat) lets git work on all images. Confirmed: sklearn-14629 gold resolved False->True (27/27 required, 0 bad-config errors). Applied across all 5 harnesses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

…sted parity) The docker gold census reaches 493/500 after the GIT_CONFIG_GLOBAL fix (445 -> 486 with -rA -> 493 with the git-env fix), matching the apptainer/.sif nested reference 492/500 to within environment noise. Documents the docker<->apptainer parity (both use the flat grader) and the two reconstruction gaps the census surfaced + closed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

Add responses_api_agents/anyswe_agent/gold_census.py, a standalone gold-patch census driver (no model, no agent) that feeds each SWE-bench instance's gold patch through the flat grader and tallies resolves -- the tool that produces the README baseline. Resumable + checkpointed; docker --rmi bounds disk. The apptainer path injects --writable-tmpfs to mirror anyswe's grading provider (the read-only base .sif otherwise fails the eval's git checkout / patch apply, scoring every instance unresolved). Reference it from the swe_env README's Reproduce section. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

The apptainer provider bind-mounts the host $HOME by default, so in-container matplotlib reads the host's matplotlib config + font cache. That flips image- comparison tests (e.g. test_axes.py::test_pcolormesh_small[eps]) vs docker: an 8-instance matplotlib gap where the gold patch resolved on docker-flat but failed on apptainer-flat. swebench's own nested .sif path avoided this with --no-mount. Add --no-mount home to the apptainer grading + agent sandboxes (_grading_provider, _provider) and to gold_census.py, alongside --writable-tmpfs. Validated: matplotlib- 24149 768/768 (was 767/768); all 8 matplotlib misses resolve under the fix, giving exact docker==apptainer parity on the graded set. --cleanenv / --no-mount tmp,bind- paths were too aggressive (broke the eval's conda/PATH env -> empty test log); HOME= /root alone is insufficient (apptainer reasserts HOME). Add 3 unit tests (19 pass). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

…efault 1800s) flat_run_eval ran the eval command with timeout_s=task.metadata.get('tests_timeout') -> None when unset -> the eval inherited each sandbox provider's exec default. Those differ enormously: docker 3600s vs apptainer 180s. So any test suite taking >180s (scikit-learn, sympy, or any suite slowed under concurrency) was silently masked as a 'timeout' on apptainer while resolving on docker -- a spurious docker<->apptainer grading gap that has nothing to do with the patch. Evidence: a tmpfs-vs-disk overlay A/B made no difference (both hit ~180s); the cap, not I/O, was the wall. Default to 1800s (matches swe_rebench's harness, anyswe's swebench_tests_timeout, and swebench run_evaluation's own per-repo budget) so the per-command budget is identical on every backend. Add 2 regression tests asserting the eval command carries 1800s (not None) and that an explicit tests_timeout still overrides. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

Follow-up to the flat_eval timeout fix: close the same gap in the rest of the library so the fix is complete, not just on the SWE-bench-Verified flat path. - verify_task: propagate the caller's eval_timeout_s into task.metadata['tests_timeout'] when unset, so the configured budget reaches the in-sandbox eval command itself (not just the outer wall-clock guard). Otherwise raising eval_timeout_s/swebench_tests_timeout had no effect on the command, which fell back to the provider exec default. - swe_bench_ext + nv_internal: run the eval command with timeout_s=tests_timeout (default 1800s), matching flat_eval + swe_rebench. Previously they passed no timeout, so the command inherited the provider exec default -- apptainer 180s vs docker 3600s -- masking long suites (scikit-learn/sympy/etc.) as timeouts on apptainer only. (r2egym already routes through flat_eval, so it inherits the fix.) - 3 verify_task regression tests (default 1800s budget; eval_timeout_s propagates to the command; explicit tests_timeout preserved). 163 swe_env tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

Rewrite the reward-profiling baseline with the docker-flat / apptainer-flat / official-nested numbers and an evidence-backed overcount analysis: - docker-flat 493/500 and apptainer-flat 490/500 use the identical host-side grader; apptainer never over-resolves. The 3-instance gap is non-grading: psf/requests-1724/2317 (external httpbin.org returning 503 at run time -- docker fails them identically when httpbin is down) and scikit-learn-14710 (eval-timeout under --concurrency 12; resolves in isolation -- verified). - 493 is NOT an overcount: it exceeds official run_evaluation 4.1.0 (490) because that tool runs sphinx-via-tox pytest without -rA and undercounts sphinx-8595/9711 (genuinely passing, tox exit 0) -- which flat correctly resolves. flat is in fact conservative (misses sphinx-8120/8265/8269). ~496/500 truly resolvable; the only genuine gold failures are astropy-7606/8707/8872 + django-10097 (real fail, confirmed once the 1800s timeout let its ~1900-test suite finish). - Document the two apptainer parity fixes (--no-mount home, 1800s eval-command timeout) and that gold_census.py applies them automatically. Add the run_evaluation cross-check command with its -rA/httpbin caveats. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

The 492 reference was swebench's NESTED run_evaluation on .sif (a different grader), not this flat grader. Spell out that docker-flat and apptainer-flat are the same grader (miss the identical 7) and both reach 493; apptainer's measured count is lower only by transient run-time conditions: psf/requests-1724/2317 hit httpbin.org while it was returning 503 (docker fails them identically when down), and sklearn-14710 hit a CONC=12 eval timeout (re-graded in isolation it resolves -> folded in, 490->491). Grading parity is exact; the gap is never the grader. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

…t parity Make the documented tool able to reproduce the FULL apptainer census without pre-staging 500 .sif files, and to run both providers under identical conditions: - --apptainer-build: build each missing .sif on-demand from docker://swebench/... (apptainer pull --disable-cache); with --rmi only the on-demand-built sifs are deleted afterward (pre-existing sifs are kept). Smoke: docker 2/2, apptainer 2/2, built sifs auto-removed. - --tests-timeout: pass a per-eval budget (default 1800) so heavy suites aren't masked by a concurrency-induced timeout when comparing providers for exact parity. - README: document running both providers in the same httpbin window + --tests-timeout 3600 for an identical passing set. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

…each) Re-ran both providers via gold_census.py in one window (same httpbin.org state) with --tests-timeout 3600 to remove the two non-grading confounds (external httpbin + a concurrency-induced eval timeout). Result: docker-flat and apptainer-flat resolve the IDENTICAL 493/500 and miss the IDENTICAL 7 (astropy-7606/8707/8872, django-10097, sphinx-8120/8265/8269) -- same passing set, same failing set, zero over-resolves either way. The earlier 493-vs-491 split was purely httpbin.org flapping 503->200 mid-run (the psf/requests instances graded at different moments on each provider) + one sklearn CONC=12 timeout; reconciling the 8 psf ids on both providers in the same window with httpbin up resolves all of them on both. Update the baseline table + state the parity-measurement recipe (one window, --tests-timeout 3600). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

…=493 The gold_census.py docstring still cited 'apptainer/.sif 492/500' (the OLD nested run_evaluation reference); update it to the verified flat result (docker-flat == apptainer-flat == 493/500, identical set). Tighten the Fern SWE-Environment note from 'equivalent resolve rates' to the verified identical 493/500 set, and point at the README for the reproduce recipe. README itself already carries the verified numbers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>

New nemo_gym/sandbox/providers/enroot: an EnrootProvider implementing the SandboxProvider Protocol via the enroot CLI (import/create/start --rw/remove), registered as the "enroot" builtin. Scheduler-agnostic (no Slurm/pyxis/path assumptions; ENROOT_* read from env). Unlike apptainer it needs no writable overlay (enroot rootfs is writable) and no host-$HOME isolation. anyswe_agent._provider: add an enroot branch injecting the per-instance binds (/trajectories_mount, /nemo_gym_mount, /agent_deps_mount) into exec.default_binds, parity with the apptainer/docker branches. anyswe_openclaw.yaml: sandbox_provider {enroot}, .sqsh container_formatter, nemotron-3-ultra. Lets OpenClaw+SWE-bench (PR NVIDIA-NeMo#1802) run on enroot-only clusters. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The committed config (from NVIDIA-NeMo#1802) hardcoded model nvinf/nvidia/qwen/qwen3.6-35b-a3b and set the OpenClaw workspace to /testbed. Against a serve that does not serve that model, OpenClaw gets no responses and emits only its bootstrap scaffolding, and workspace=/testbed makes that scaffolding pollute the graded diff. Points the model at ${policy_model_name} (follows run_eval's -m flag), moves OpenClaw's workspace and run dir off /testbed (workspace_root /tmp/openclaw_ws, workspace .) to keep the /testbed diff clean, and adds node_bin_dir /agent_deps_mount/bin so the portable Node 22 wins over a task image's Node 18. Validated on a 5-task smoke (nemotron-nano-3.5-ea2): 3/5 resolved with genuine repo edits, 0 scaffolding pollution, 0 node failures. Full verified500 run in progress. Draft for review.

codex and others added 5 commits June 28, 2026 00:54

adil-a requested a review from a team as a code owner June 28, 2026 00:55

codex and others added 2 commits June 28, 2026 01:40

ffrujeri self-requested a review June 28, 2026 02:45

ffrujeri approved these changes Jun 28, 2026

View reviewed changes

codex and others added 9 commits June 28, 2026 02:57

adil-a merged commit 4aa34fc into NVIDIA-NeMo:cmunley1/anyswe Jun 29, 2026
13 of 15 checks passed

j-nolan mentioned this pull request Jul 19, 2026

anyswe_openclaw: run OpenClaw against /testbed with the run_eval model elisam0/Gym#9

Merged

Navigation Menu

Uh oh!

swe_env/anyswe: address @ffrujeri review on #1572 (fixes + gold baseline) - #1802

swe_env/anyswe: address @ffrujeri review on #1572 (fixes + gold baseline)#1802
adil-a merged 16 commits into
NVIDIA-NeMo:cmunley1/anyswefrom
adil-a:swe-env-review-fixes

adil-a commented Jun 28, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jun 28, 2026

Uh oh!

github-actions Bot commented Jun 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

adil-a commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review-comment fixes (@ffrujeri)

Gold-patch baseline (review #4) — at parity with the nested reference

Validation

Uh oh!

copy-pr-bot Bot commented Jun 28, 2026

Uh oh!

github-actions Bot commented Jun 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

adil-a commented Jun 28, 2026 •

edited

Loading