feat(swe_env): decoupled SWE environment library + verifier (#1249) [1/3] - #1677
Closed
adil-a wants to merge 1 commit into
Closed
feat(swe_env): decoupled SWE environment library + verifier (#1249) [1/3]#1677adil-a wants to merge 1 commit into
adil-a wants to merge 1 commit into
Conversation
Adds the provider-neutral swe_env library (sandbox lifecycle, docker/apptainer providers, per-benchmark task harnesses, test-output parsing, grading) and the required stateless verifier resources server that grades an agent's patch in a fresh sandbox and returns the reward. No agent is wired to it yet; it stands alone and is covered by its own unit tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
adil-a
changed the title
This was referenced Jun 23, 2026
Closed
2 tasks
adil-a
pushed a commit
to adil-a/Gym
that referenced
this pull request
Jun 26, 2026
…b + verifier + docker provider; drop openclaw) Self-contained anyswe-on-swe_env on top of cmunley1/anyswe (now merged with latest main): - add swe_env library (responses_api_agents/swe_env/) + inline verifier (resources_servers/swe_env/) - add the docker sandbox provider + register it (main has apptainer/opensandbox, not docker) - replace anyswe_agent with the swe_env-based version: docker + apptainer backends, local-.sif support, --writable-tmpfs, claude_code anthropic_base_url=null (real Anthropic by default) - drop openclaw + orphaned dataset setup scripts + unused example_rollouts (out of scope) Supersedes the NVIDIA-NeMo#1677 stack (NVIDIA-NeMo#1677/NVIDIA-NeMo#1678/NVIDIA-NeMo#1679) and the prior NVIDIA-NeMo#1738 base (stack/3). Unit tests: anyswe 16, swe_env lib 134 (+1 skip), verifier 14 (+2 skip) = 164 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Author
adil-a
added a commit
that referenced
this pull request
Jun 27, 2026
…erified (#1249) (#1738) ## What this is Converges the **anyswe** runner (#1572) onto the **#1677 swe_env decoupling**: a thin agent drops any Gym agent into the SWE task container, extracts `git diff HEAD`, and grades **in-process** via `verify_task` — no separate resources server. Based on the #1677 stack, so this diff is just the convergence + fixes (~55 files). ## Before / after ```text BEFORE (main) task --> swe_agents (OpenHands-coupled; sandbox + grading inline; apptainer-only) AFTER (this PR) [NEW] = added in this PR task --> anyswe_agent --> swe_env/ [NEW shared library] (any Gym agent +-- harnesses/ (extracted from swe_agents) in the container) +-- parsing/ (verbatim from main) +-- verify_task (inline grade; no /verify server) | v nemo_gym/sandbox providers: docker [NEW] | apptainer ``` ## What changed | Path | Change | |---|---| | `responses_api_agents/swe_env/` | **New** shared library — `harnesses/`, `parsing/`, sandbox lifecycle, `verify_task`, `self_drive` | | `responses_api_agents/anyswe_agent/` | Thin runner (#1572) rewired onto `swe_env`; grades `git diff` in-process | | `nemo_gym/sandbox/providers/docker/` | **New** docker provider (was apptainer-only) — runs Verified on docker | | `…/swe_env/parsing/` | Byte-identical relocation of main's `swe_agents/swe_bench_ext/` (lighthouse parser) | | `…/swe_env/verify_task.py` | Moved from `resources_servers/swe_env/`; inline grader, importers repointed | | `resources_servers/swe_env/` | **Removed** — no consumer; grading is inline, not a `/verify` server | | `…/anyswe_agent/setup_scripts/{swebench,r2e_gym,swebench_multilingual}.sh` | **Removed** — superseded by `harnesses/` + pip `swebench` | | `responses_api_agents/openclaw_agent/` | Untouched (a `main` artifact — neither added nor removed) | Other fixes: docker flat-eval uses swebench's official per-repo parser (gold resolve 4%→95%); per-repo FAIL_ONLY eval-type + main's patch-apply ladder ported into the harnesses; apptainer `--pid` removed (≥1.3.6 rejects it); `get_harness` self-registers + HF-name tolerant. ## Suggested review order 1. `nemo_gym/sandbox/providers/registry.py` + `providers/docker/` — the new provider lazy-registration (small, foundational). 2. `responses_api_agents/swe_env/harness.py` — the contract: `SweTask`, the harness ABC, the registry, and the resolution rule (`compute_resolved`). **Read first** — everything else builds on it. 3. `responses_api_agents/swe_env/verify_task.py` — the inline grading orchestrator (the "no `/verify` server" core). 4. `responses_api_agents/swe_env/sandbox.py` + `self_drive.py` — sandbox lifecycle, then provision / egress / `git diff` extraction. 5. `responses_api_agents/swe_env/harnesses/swebench.py` — the primary harness + the parity fixes (FAIL_ONLY, patch-apply ladder, masking); then `swe_bench_ext.py` / `r2egym.py` / `flat_eval.py` as needed. 6. `responses_api_agents/anyswe_agent/app.py` — how it wires together: routing → `_build_swetask` → provider → `self_drive` + `verify_task`. 7. `responses_api_agents/swe_env/parsing/` — **skim only**: byte-identical relocation from main, no new logic. 8. `…/swe_env/tests/` + `…/anyswe_agent/tests/` — coverage for the above. ## Validation — SWE-bench Verified (N=100/cell), in-line grader Each agent × model run on the **same 100 instances** on **both** sandbox backends; cells are resolve rate (`mean/reward`), all clean (mask ≤ 0.01): | agent × model | docker | apptainer | |---|---|---| | hermes × gpt-5.5 | 69% | 66% | | hermes × opus-4.8 | 77% | 79% | | claude_code × opus-4.8 | 78% | 82% | | openclaw × gpt-5.5 | 70% | 71% | | openclaw × opus-4.8 | 77% | 78% | docker and apptainer agree within ±4 pts (mean Δ ≈ +1, no systematic gap) — the two backends are equivalent. (claude_code × gpt-5.5 skipped: the Claude CLI speaks Anthropic `/v1/messages` only.) Gold-patch grader sanity beforehand: docker **95/100**, `.sif` **10/10**. - A full per-failure audit found **0 our-side (infra) failures** — non-resolved rows are model wrong/no-patch. (Found + fixed an apptainer ENOSPC en route: `--writable-tmpfs`'s 64 MiB cap → per-instance disk `--overlay`.) - ruff clean, **158 unit tests pass**; harness-vs-main audit converged (0 residual behavioral drift). ## Reproduce — gpt-5.5 on hermes (docker **or** apptainer) Shared setup + dataset (gpt-5.5 is a reasoning model, so temperature/top-p must be 1.0): ```bash uv venv && uv sync --extra dev export RAY_TMPDIR=/tmp # avoids Ray AF_UNIX path-length limits export POLICY_API_KEY=sk-... # gateway key — SECRET, pass via ++ only, never commit python responses_api_agents/anyswe_agent/prepare.py --no-images --limit 100 --temperature 1.0 --top-p 1.0 DS=responses_api_agents/anyswe_agent/data/swebench_verified.jsonl MODEL=us/azure/openai/eccn-gpt-5.5 A=anyswe_hermes.responses_api_agents.anyswe_agent ``` **Docker** (default provider; each instance image pulls on demand from DockerHub — no pre-build): ```bash # Terminal A — servers (wait for "All 2 / 2 servers ready!") ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" \ ++policy_base_url='https://inference-api.nvidia.com/v1' ++policy_model_name="$MODEL" ++policy_api_key="$POLICY_API_KEY" # Terminal B — rollouts (num_samples_in_parallel == concurrent containers) ng_collect_rollouts +agent_name=anyswe_hermes +input_jsonl_fpath=$DS \ +output_jsonl_fpath=results/hermes_gpt_docker.jsonl +num_samples_in_parallel=16 ``` **Apptainer** (build a `.sif` per instance once, then run with no registry pulls): ```bash # Build SIFs (needs apptainer on PATH; ~1-3 GB each) -> data/sifs/{instance_id}.sif python responses_api_agents/anyswe_agent/prepare.py --no-dataset --sif-dir data/sifs # Terminal A — same servers, but point the formatter at the .sif and select the apptainer provider. # Quote container_formatter so Hydra treats {instance_id} as a literal; sandbox_provider is a dict. ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" \ ++policy_base_url='https://inference-api.nvidia.com/v1' ++policy_model_name="$MODEL" ++policy_api_key="$POLICY_API_KEY" \ ++$A.container_formatter='"data/sifs/{instance_id}.sif"' \ ++$A.sandbox_provider='{apptainer:{}}' # Terminal B — same collect command ng_collect_rollouts +agent_name=anyswe_hermes +input_jsonl_fpath=$DS \ +output_jsonl_fpath=results/hermes_gpt_appt.jsonl +num_samples_in_parallel=16 ``` Resolve rate = `mean/reward` in `results/<name>_aggregate_metrics.json`; a clean run has `mean/mask_sample == 0`. (For other cells: `anyswe_claude_code.yaml` + an opus model id speaks Anthropic `/v1/messages` via `++$A.agent_kwargs.anthropic_base_url=…`; `anyswe_openclaw.yaml` runs OpenClaw. Same `container_formatter` / `sandbox_provider` overrides switch either between docker and apptainer.) ## Deferred to a follow-up openclaw eval (runner restored, eval deferred); R2E-Gym / Multilingual **native nested graders** (host-side flat grading approximates those — Verified/English is exact); routing needles for `swe-bench-ext` / `swe-rebench` / `nv-internal`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Codex <codex@openai.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Decoupled SWE environment library + verifier (#1249)
Stacked PR 1 of 3 · base:
main· next:stack/2-swe-agents-openhandsPart of #1249 — decouple the SWE environment infrastructure from the agent harnesses — built on top of the Sandbox API from #1377 (now on
main).What this PR adds
Today the entire SWE-bench stack (sandbox provisioning, the per-dataset eval recipes, patch parsing, and grading) is fused into the OpenHands agent in
responses_api_agents/swe_agents/app.py. No other agent can reuse any of it. This PR extracts that infrastructure into two reusable, agent-agnostic pieces. It adds the new pieces only — no existing agent is rewired yet (that is PR 2).1.
responses_api_agents/swe_env/— a shared SWE environment library. An installable distribution that any agent imports to provision and drive its own task sandbox on top of the #1377 Sandbox API:environment.py—AsyncSweEnvironment, the provider-neutral execute/upload/download wrapper agents drive.harness.py/registry.py— theSweTaskHarnessABC and a name→harness registry; one source of truth for the dataset families (swebench, swe-rebench, nv-internal, swe-bench-ext, r2egym, flat-eval).grading.py— purecompute_resolved()/reward_from_report()helpers.parsing/— the SWE-bench-Ext test-log parser (relocated here as the single home; the duplicate underswe_agentsis removed in PR 2).providers/{apptainer,docker}_provider.py—SandboxProviderimplementations, including the Apptainer.sifpath the on-prem clusters use.lifecycle.py— a thinacquire_sandbox()context manager. (The heavier durable-registry/reaper/admission lifecycle layer is intentionally out of scope here and tracked as a follow-up to generalize it into the sandbox layer.)model_endpoint.py— provider-neutral resolution of a sandbox-reachable model-server endpoint for self-driving agents.2.
resources_servers/swe_env/— the verifier. ASimpleResourcesServerwhoseverify()is the single scoring entry point: it extracts the model's patch from the response, grades it in a fresh, stateless sandbox, and returns the eval-side fields.verify_task.pyholds the server-private grading orchestrator (reset → apply patch → run eval → parse → reward). Ships with a launchable config + offline example fixtures for the data gate.Tests
ruff check/ruff formatclean. Real-container grading paths are skip-guarded (apptainer/Docker); the orchestrator is exercised via a FakeSandbox that replays recorded eval transcripts.Review notes
requirements.txt), so the cross-tree import works underng_run/ng_test.🤖 Generated with Claude Code
Stack created with GitHub Stacks CLI • Give Feedback 💬