Converge #1677 swe_env decoupling into anyswe + verify on SWE-bench Verified (#1249)

[adil-a]

What this is

Converges the anyswe runner (#1572) onto the #1677 swe_env decoupling: a thin agent drops any Gym agent into the SWE task container, extracts git diff HEAD, and grades in-process via verify_task — no separate resources server. Based on the #1677 stack, so this diff is just the convergence + fixes (~55 files).

Before / after

BEFORE (main)

  task --> swe_agents     (OpenHands-coupled; sandbox + grading inline; apptainer-only)


AFTER (this PR)                                       [NEW] = added in this PR

  task --> anyswe_agent --> swe_env/   [NEW shared library]
             (any Gym agent       +-- harnesses/    (extracted from swe_agents)
              in the container)    +-- parsing/      (verbatim from main)
                                   +-- verify_task   (inline grade; no /verify server)
                                          |
                                          v
                      nemo_gym/sandbox providers:   docker [NEW]  |  apptainer

What changed

Path	Change
responses_api_agents/swe_env/	New shared library — harnesses/, parsing/, sandbox lifecycle, verify_task, self_drive
responses_api_agents/anyswe_agent/	Thin runner (#1572) rewired onto swe_env; grades git diff in-process
nemo_gym/sandbox/providers/docker/	New docker provider (was apptainer-only) — runs Verified on docker
…/swe_env/parsing/	Byte-identical relocation of main's swe_agents/swe_bench_ext/ (lighthouse parser)
…/swe_env/verify_task.py	Moved from resources_servers/swe_env/; inline grader, importers repointed
resources_servers/swe_env/	Removed — no consumer; grading is inline, not a /verify server
…/anyswe_agent/setup_scripts/{swebench,r2e_gym,swebench_multilingual}.sh	Removed — superseded by harnesses/ + pip swebench
responses_api_agents/openclaw_agent/	Untouched (a main artifact — neither added nor removed)

Other fixes: docker flat-eval uses swebench's official per-repo parser (gold resolve 4%→95%); per-repo FAIL_ONLY eval-type + main's patch-apply ladder ported into the harnesses; apptainer --pid removed (≥1.3.6 rejects it); get_harness self-registers + HF-name tolerant.

Suggested review order

1. nemo_gym/sandbox/providers/registry.py + providers/docker/ — the new provider lazy-registration (small, foundational).
2. responses_api_agents/swe_env/harness.py — the contract: SweTask, the harness ABC, the registry, and the resolution rule (compute_resolved). Read first — everything else builds on it.
3. responses_api_agents/swe_env/verify_task.py — the inline grading orchestrator (the "no /verify server" core).
4. responses_api_agents/swe_env/sandbox.py + self_drive.py — sandbox lifecycle, then provision / egress / git diff extraction.
5. responses_api_agents/swe_env/harnesses/swebench.py — the primary harness + the parity fixes (FAIL_ONLY, patch-apply ladder, masking); then swe_bench_ext.py / r2egym.py / flat_eval.py as needed.
6. responses_api_agents/anyswe_agent/app.py — how it wires together: routing → _build_swetask → provider → self_drive + verify_task.
7. responses_api_agents/swe_env/parsing/ — skim only: byte-identical relocation from main, no new logic.
8. …/swe_env/tests/ + …/anyswe_agent/tests/ — coverage for the above.

Validation — SWE-bench Verified (N=100/cell), in-line grader

Each agent × model run on the same 100 instances on both sandbox backends; cells are resolve rate (mean/reward), all clean (mask ≤ 0.01):

agent × model	docker	apptainer
hermes × gpt-5.5	69%	66%
hermes × opus-4.8	77%	79%
claude_code × opus-4.8	78%	82%
openclaw × gpt-5.5	70%	71%
openclaw × opus-4.8	77%	78%

docker and apptainer agree within ±4 pts (mean Δ ≈ +1, no systematic gap) — the two backends are equivalent. (claude_code × gpt-5.5 skipped: the Claude CLI speaks Anthropic /v1/messages only.) Gold-patch grader sanity beforehand: docker 95/100, .sif 10/10.

• A full per-failure audit found 0 our-side (infra) failures — non-resolved rows are model wrong/no-patch. (Found + fixed an apptainer ENOSPC en route: --writable-tmpfs's 64 MiB cap → per-instance disk --overlay.)
• ruff clean, 158 unit tests pass; harness-vs-main audit converged (0 residual behavioral drift).

Reproduce — gpt-5.5 on hermes (docker or apptainer)

Shared setup + dataset (gpt-5.5 is a reasoning model, so temperature/top-p must be 1.0):

uv venv && uv sync --extra dev
export RAY_TMPDIR=/tmp                  # avoids Ray AF_UNIX path-length limits
export POLICY_API_KEY=sk-...            # gateway key — SECRET, pass via ++ only, never commit
python responses_api_agents/anyswe_agent/prepare.py --no-images --limit 100 --temperature 1.0 --top-p 1.0
DS=responses_api_agents/anyswe_agent/data/swebench_verified.jsonl
MODEL=us/azure/openai/eccn-gpt-5.5
A=anyswe_hermes.responses_api_agents.anyswe_agent

Docker (default provider; each instance image pulls on demand from DockerHub — no pre-build):

# Terminal A — servers (wait for "All 2 / 2 servers ready!")
ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" \
  ++policy_base_url='https://inference-api.nvidia.com/v1' ++policy_model_name="$MODEL" ++policy_api_key="$POLICY_API_KEY"
# Terminal B — rollouts (num_samples_in_parallel == concurrent containers)
ng_collect_rollouts +agent_name=anyswe_hermes +input_jsonl_fpath=$DS \
  +output_jsonl_fpath=results/hermes_gpt_docker.jsonl +num_samples_in_parallel=16

Apptainer (build a .sif per instance once, then run with no registry pulls):

# Build SIFs (needs apptainer on PATH; ~1-3 GB each) -> data/sifs/{instance_id}.sif
python responses_api_agents/anyswe_agent/prepare.py --no-dataset --sif-dir data/sifs
# Terminal A — same servers, but point the formatter at the .sif and select the apptainer provider.
# Quote container_formatter so Hydra treats {instance_id} as a literal; sandbox_provider is a dict.
ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/openai_model/configs/openai_model.yaml]" \
  ++policy_base_url='https://inference-api.nvidia.com/v1' ++policy_model_name="$MODEL" ++policy_api_key="$POLICY_API_KEY" \
  ++$A.container_formatter='"data/sifs/{instance_id}.sif"' \
  ++$A.sandbox_provider='{apptainer:{}}'
# Terminal B — same collect command
ng_collect_rollouts +agent_name=anyswe_hermes +input_jsonl_fpath=$DS \
  +output_jsonl_fpath=results/hermes_gpt_appt.jsonl +num_samples_in_parallel=16

Resolve rate = mean/reward in results/<name>_aggregate_metrics.json; a clean run has mean/mask_sample == 0. (For other cells: anyswe_claude_code.yaml + an opus model id speaks Anthropic /v1/messages via ++$A.agent_kwargs.anthropic_base_url=…; anyswe_openclaw.yaml runs OpenClaw. Same container_formatter / sandbox_provider overrides switch either between docker and apptainer.)

Deferred to a follow-up

openclaw eval (runner restored, eval deferred); R2E-Gym / Multilingual native nested graders (host-side flat grading approximates those — Verified/English is exact); routing needles for swe-bench-ext / swe-rebench / nv-internal.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ffrujeri]

Where this PR sits

flowchart LR
    DS[(Dataset JSONL<br/>SWE-bench Verified)] --> Anyswe[anyswe_agent/app.py<br/>thin runner]
    Anyswe -->|_build_swetask| Harness[swe_env/harness.py<br/>SweTask + ABC]
    Anyswe -->|drive inner agent| SelfDrive[swe_env/self_drive.py]
    SelfDrive <-->|chat/responses| Model[Model Server]
    Anyswe -->|acquire_sandbox| Sandbox[swe_env/sandbox.py]
    Sandbox --> Docker[providers/docker NEW]
    Sandbox --> Appt[providers/apptainer]
    Anyswe -->|git diff HEAD| Verify[swe_env/verify_task.py<br/>inline grader]
    Verify --> Grade[harness.grade + compute_resolved<br/>reward 1.0/0.0]
    Grade -.->|reward| RL[(Nemo-RL / eval)]

Loading

This converges the anyswe runner onto the swe_env decoupling: grading moves from a separate /verify resources server to an inline verify_task library call that re-grades the agent's patch in a fresh, hermetic sandbox, plus a new docker sandbox provider (was apptainer-only) and a shared swe_env/ library (harnesses, parsing, lifecycle, grading). Stacked on the unmerged base cmunley1/anyswe.

Really nice work — this is a substantial, well-structured collapse (swe_env 9→5 files), and a lot held up under review: /run is async with exception isolation (reward is always a non-null float), sandbox teardown is guaranteed on every path (no container leak), and compute_resolved checks out as line-by-line equivalent to canonical swebench's check_pass_and_fail / check_fail_only / get_resolution_status (including the per-repo FAIL_ONLY_REPOS selection and the verbatim git apply → git apply --reject → patch --fuzz=5 ladder). The suite runs green and lint-clean locally (158 passing).

The inline comments are mostly questions — a couple of possible regressions to sanity-check (model_patch recording, a masking case vs main), a reproducibility question (pinning swebench), some docker-provider create()/concurrency hardening, and a few test-coverage questions. None are blockers.

[ffrujeri]

app.py:607

Was dropping model_patch from the metrics dict here intentional? On the base branch metrics.model_patch = patch or None was set, so as written AnySweVerifyResponse.model_patch comes back None even when patch_exists=True. Reward is unaffected (it's derived from resolved), but would downstream consumers that read the patch from the structured result (rollout viewer / BLADE / training) still get what they need, given the patch now survives only as the on-disk patch.diff? If it should be recorded, would something like this work?

Suggested change

	update_metrics(
	update_metrics(
	params.metrics_fpath,
	{
	"resolved": resolved,
	"patch_exists": bool(patch.strip()),
	"model_patch": patch or None,
	"agent_timed_out": agent_timed_out,
	"error_kind": report.error_kind,
	"mask_sample": params.mask_sample,
	"openhands_run_time": agent_run_time,
	},
	)

[adil-a]

Thanks for the catch, adding it in

[ffrujeri]

app.py:605

Question on the masking here: this masks only on error_kind / agent_timed_out, whereas main's swe_agents grading also masks the case where a task is resolved and the agent hit max_iteration / context_window (the “patch happened to pass but the agent never properly submitted / blew its context” accidental-reward mask). Is dropping that mask intentional for this convergence? It's negligible for the eval resolve-rate, but for training rollouts those accidental passes would leak into the gradient at reward 1.0 — curious whether that's acceptable here.

[ffrujeri]

app.py:389

A question about the patch the runner extracts: _RUNNER_TEMPLATE captures the diff with git diff HEAD, which excludes untracked / newly-created files. Since patch.diff is always written, the git add -A && git diff --cached fallback in self_drive looks like it never runs — so if an agent resolves an issue by adding a new file/module, would that file be omitted from the graded patch (a possible false-negative)? This seems carried over from the base branch rather than introduced here, but since grading fidelity on SWE-bench Verified is the goal of this PR, would it be worth a git add -A before the diff so capture matches SWE-bench's own model-patch extraction (gold patches do include new files)?

[adil-a]

Good catch! :D I can add it in

[ffrujeri]

requirements.txt:4

Would it be worth pinning swebench here? The resolve verdict now depends on swebench's per-repo log parsers, FAIL_ONLY_REPOS, and eval-script templates, all of which change across releases — so a bare swebench could let the grade drift across installs and over time. Could it be pinned to the version that produced the reported 95/100 + agent matrix (e.g. swebench==4.1.0, matching mini_swe_agent_2) to keep the numbers reproducible?

Suggested change

	swebench
	swebench==4.1.0

[ffrujeri]

swebench.py:226

When the swebench import (or make_test_spec / get_logs_eval) fails, grading falls back to the generic pytest-only parser, which — per this file's own docstring — mis-grades non-pytest repos like django as unresolved even for a correct patch. Combined with the unpinned swebench dep, could a bad or mismatched install quietly degrade SWE-bench Verified scoring rather than surfacing? Would it be safer to fail loud for swe-bench instances when swebench is unavailable, instead of degrading to the generic parser?

[ffrujeri]

provider.py:77

Would it be worth bounding _run with an asyncio.Semaphore here? ApptainerProvider bounds its docker/apptainer subprocesses with self._semaphore (CLAUDE.md: “bound concurrent subprocess/external calls”), but the docker provider spawns CLI processes unbounded. Concurrency is bounded at the call site today, so it isn't a blocker — but would a configurable per-provider semaphore be a useful match to the apptainer baseline on the SWE hot path (one sandbox per rollout, many exec each)?

[ffrujeri]

provider.py:165

Two small questions about create() robustness:

1. If docker run -d returns rc 0 with empty stdout, would "".splitlines()[-1] raise IndexError before the intended if not container_id guard on the next line — i.e. is that guard currently reachable? Guarding the empty case would keep the friendly SandboxCreateError.
2. The docker run -d has no pre-assigned --name/label — on a timeout that kills the CLI client after the daemon already started the container, would that orphan be reachable by teardown (no handle captured yet)? Apptainer pre-generates a uuid name so a failed start stays cleanable; would a --name <uuid> close that gap here? Something like:

Suggested change

	container_id = out.strip().splitlines()[-1].strip()
	lines = out.strip().splitlines()
	container_id = lines[-1].strip() if lines else ""
	if not container_id:
	raise SandboxCreateError("docker run did not return a container id")

[ffrujeri]

test_swe_env.py:126

Two questions on the grading-contract coverage:

1. The harness.py module docstring states “a test asserts agent adapters never reference reset_repo / run_eval / grade”, but I couldn't find that test — does it exist somewhere, or would it be worth adding an AST/source-inspection guard (or softening the docstring)? The boundary does hold by inspection today.
2. test_compute_resolved exercises the no-status_map passed-set fallback — would it be worth also covering the status_map pass_and_fail path (the one that runs for SWE-bench Verified: neutral SKIPPED/XPASS exclusion, ERROR/absent → fail)? A populated-status_map case would lock in the swebench-equivalence this PR depends on.

[ffrujeri]

test_flat_eval.py:607

Since test_flat_vs_nested_equivalence_on_gold unconditionally raise AssertionError(...), it will fail (not skip) for anyone who sets SWE_ENV_RUN_REAL_CONTAINERS=1 — would pytest.skip("equivalence harness not yet implemented") in the body read better as a known-TODO? (Unrelated: test-agent verified 3 small tests that close the verify_task.py coverage gaps — the flat_eval metadata branch, ttl stamping, and the report_to_reward wrapper — happy to share them if they'd be useful.)