feat: decouple SWE environment infra from agent harnesses (#1249) - #1
Closed
adil-a wants to merge 35 commits into
Closed
feat: decouple SWE environment infra from agent harnesses (#1249)#1adil-a wants to merge 35 commits into
adil-a wants to merge 35 commits into
Conversation
…VIDIA-NeMo#1249) First increment of NVIDIA-NeMo#1249 on top of the NVIDIA-NeMo#1377 Sandbox API: a provider-neutral responses_api_agents/swe_env library (provisioning + exec + per-family harness recipes + grading) and a required, stateless, fresh-only resources_servers/swe_env verifier with a server-private verify_task orchestrator. - swe_env library: SweTask/harness contract (provisioning vs server-private grading split), AsyncSweEnvironment over nemo_gym.sandbox, registry, pure grading helpers, swe-bench-ext reference harness. - providers: DockerSandboxProvider (real/local, enables end-to-end testing without apptainer) + ApptainerSandboxProvider (ports the legacy .sif path; mocked-tested). - verifier: SweEnvVerifier.verify() extracts the patch from the response, grades in its own fresh sandbox, masks infra failures as reward=0.0 (never None). - 25 tests incl. an env-gated real docker-backed end-to-end (gold patch -> resolved). See SWE_ENV_DECOUPLE_STATUS.md for scope, what's tested, and follow-ups. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
adil-a
force-pushed
the
feat/swe-env-decouple-1249
branch
from
June 19, 2026 08:28
c3a03ae to
8498322
Compare
- Add requirements.txt to both servers (-e nemo-gym[dev] @ ../../) so the isolated per-server ng_test venv installs the project editable and the cross-tree imports resolve (mirrors mini_swe_agent_2). - Restore .gitignore to base (local-only ignores moved to .git/info/exclude) so the PR touches only server files -> CI runs per-server ng_test (the data gate / full suite is reserved for 'other' changes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…reaper/idempotency Item 2 (families) + item 3 (lifecycle) of the NVIDIA-NeMo#1249 plan: - Relocate the vendored swe_bench_ext parser (1606 lines) into swe_env/parsing/. - Add 5 family harnesses: nv_internal + swe_rebench (flat host-graded, run on docker), swebench (swe-bench + swe-bench-multilingual) + r2egym (nested-harness, apptainer-only, fail-fast on exec-only providers). All registered in harnesses/__init__. - Add lifecycle.py (durable SandboxRegistry + CreateAdmission + acquire_sandbox always-teardown + content_key) and reaper.py (ttl/owner-pid reaping, never reaps a live sibling, atexit bulk-stop). - Rewire verify_task to use acquire_sandbox + content-key idempotency (coalesces unbounded ServerClient retries -> one create) + per-call eval timeout (masks as eval_timeout). - 79 unit tests pass (+ real docker e2e); nested families unit-tested via FakeSandbox, real-instance validation deferred to an apptainer cluster. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…+ data-gate fixtures Item 5 (cross-cutting) of the NVIDIA-NeMo#1249 plan: - model_endpoint.py: provider-neutral in-sandbox egress primitive (§6) — apptainer loopback, opensandbox requires a cluster Service URL (else ModelEgressUnavailable), minimal env injection (never serializes the global-config dict). +tests. - Wire the SandboxReaper into the verifier server (startup/shutdown handlers; ttl/owner-pid backstop). - Add resources_servers/swe_env/configs/swe_env.yaml (verifier + agent + example dataset trio) and data-gate fixtures (example.jsonl/example_metrics.json/example_rollouts.jsonl, 5 rows) so ng_test_all passes upstream. Rollouts are synthetic gold-patch placeholders pending a real run before verified:true. - 84 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…IA-NeMo#1249 §4a) Validates the agent<->verifier contract: a standard BaseVerifyRequest (response carries the normalized model_patch) -> verifier extracts the patch, grades in a fresh sandbox, returns a non-nullable float reward (1.0 resolved / 0.0 masked) + mask_sample/eval_error. Proves any agent that POSTs a patch is scored, without coupling to a specific harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…blockers Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
adil-a
changed the title
… positive - scripts/update_env_list.py adds the swe_env resources-server row (fixes the 'Update environment list in README' lint hook). - test_model_endpoint: drop the unused api_key literal + use a variable so detect-secrets' keyword detector no longer flags the test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…1249 item 4 cutover path) swe_agents now consumes the decoupled env: swe_env_adapter.run_self_driving provisions the OpenHands working container via swe_env.lifecycle.acquire_sandbox, injects a sandbox-reachable model endpoint (egress §6), self-drives (one long exec), extracts the unified-diff patch, and scores it through the verifier in its own fresh sandbox (§4a). Additive — legacy run() untouched so test_app.py stays green; flipping run() to it (+ deleting the legacy two-container eval after a dual-run parity window) is the apptainer/OpenHands-gated final step. +FakeSandbox tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
Validated on a real apptainer 1.3.6 rootless setup: builds a .sif from the docker itest image and runs verify_task through ApptainerSandboxProvider (instance start + bind-mount IO + exec) on the swe-bench-ext flat path -> resolved=True, reward=1.0. Both providers (docker + apptainer) now proven end-to-end. Gated by SWE_ENV_APPTAINER_ITEST=1 so CI skips it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…un() flip is OpenHands/instance-gated Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
… reset_repo drops git clean Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…idated 2/2 resolved) Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…t+runs) Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…s default) Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…ress+output.jsonl paths Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…verification Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…ker path Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…val_via_verifier flag Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…ozen row + mask preserved via metrics) Signed-off-by: adil-a <adil.asif2000@hotmail.com>
… POST contract, infra-error row) Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…v reuse, NVIDIA-NeMo#1249 C10) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…ker/opensandbox); fixture parser tests Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…base.yaml
Introduce responses_api_agents/swe_agents/configs/swe_env_base.yaml as the
single source of truth for the SWE env constants that were copy-pasted across
the OpenHands-based configs (apptainer_memory_limit_mb=32768,
command_exec_timeout=300, swebench_agent_timeout=1800, and the OpenHands
agent_framework_repo/commit). The three consuming configs (swebench_openhands,
swebench_multi_tools, swebench_openhands_training) now pull each leaf in
per-key via ${inherit_from:swe_env_base.shared.*} and co-load the base via
their own config_paths (loaded transitively, so launch ergonomics are
unchanged).
Behavior is preserved byte-for-byte: every env key stays addressable at its
legacy dotted path, eval configs still resolve swebench_tests_timeout=900 and
both training blocks still resolve 1200 (left inline, intentionally not
shared). swebench_swe_agent.yaml is left untouched because it uses a different
agent framework fork and does not share these constants.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: adil-a <adil.asif2000@hotmail.com>
Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…et gated Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…ig.toml builders Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…in mini_swe (false positives) Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…mini_swe Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…secrets-clean) Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…h is the only eval (NVIDIA-NeMo#1249 A6) Signed-off-by: adil-a <adil.asif2000@hotmail.com>
… the only eval Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…plicated swe_agents/swe_bench_ext (-1946) Signed-off-by: adil-a <adil.asif2000@hotmail.com>
…dd -A Signed-off-by: adil-a <adil.asif2000@hotmail.com>
… B items Signed-off-by: adil-a <adil.asif2000@hotmail.com>
adil-a
pushed a commit
that referenced
this pull request
Jun 25, 2026
…ool error Addresses review feedback on MCPResourcesServer: - (#3) Replace the per-process token->session dict with a stateless signed token (itsdangerous URLSafeSerializer keyed by the deterministic session-middleware secret). Any worker can verify a token another worker minted, so this works with num_workers > 1 and there is nothing to evict. - (#1) Offload blocking sync @gym_tool methods to a threadpool so they don't stall the event loop (and every concurrent rollout in the worker). - (#2) Raise a plain MCPSessionError with a clean message instead of HTTPException(401). MCP runs over JSON-RPC (HTTP 200), so the status code never reaches the client; FastMCP surfaces this as a tool error (isError: true). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>
|
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. |
|
This PR was closed because it has been inactive for 7 days since being marked as stale. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Decouples SWE environment infrastructure from agent harnesses (issue NVIDIA-NeMo#1249), built on the NVIDIA-NeMo#1377 Sandbox API. Base = a copy of the NVIDIA-NeMo#1377 branch, so the diff is only new code; retarget to upstream
mainonce NVIDIA-NeMo#1377 merges.Two-piece split:
responses_api_agents/swe_env/(provider-neutral library any agent imports) +resources_servers/swe_env/(required, stateless, fresh-sandbox verifier — the sole verification entry point). The legacy two-container apptainer eval path is removed; the verifier is the only eval path.Implemented — library + verifier
swe-bench-ext/nv-internal-1/swe-rebench(flat, docker-runnable) +swe-bench/swe-bench-multilingual/r2e-gym(nested).DockerSandboxProvider(real/local) +ApptainerSandboxProvider(ports the legacy.sifpath).acquire_sandbox, reaper (ttl + owner-pid, atexit bulk-stop), content-key idempotency (coalesces unboundedServerClientretries → one create), per-call eval timeout.verify(BaseVerifyRequest)extracts the patch, grades in a fresh sandbox via server-privateverify_task, returns a non-nullable reward (masking =reward=0.0 + mask_sample, neverNone).Implemented — agent cutover (the env is now actually consumed)
swe_agents(OpenHands) cut over to the verifier.run()provisions a single working sandbox viaswe_env(_run_decoupled_agent), self-drives OpenHands, extracts the patch fromoutput.jsonl, and POSTs to the verifier (_verify_patch_via_server).resolved/eval_timed_outflow through the samemetrics_fpath, so the frozenSWEBenchVerifyResponserow +mask_sampleare preserved by construction (shared_should_mask_sample). Golden-patch verification also routed through the verifier.app.py2412 → 1351 lines (−1061): removedActiveContainerCommand,_start/_finish/_kill_container_command,_build_apptainer_command,_find_container, the sleep-until-predictions handshake, and every processor'sget_run_command.eval_via_verifieris the default and only path.mini_swe_agent_2reuses the verifier (opt-ineval_via_verifier) — proves the decoupling works for a second agent over the same contract.swe_env_base.yamlpulled per-leaf via${inherit_from:...}(evaltests_timeout=900/ train1200preserved)..sifequivalence is infra-gated).model_endpoint(+ the OpenHandsNEMO_GYM_*egress in the adapter).Testing
swe_env+swe_agents+mini_swe_agent_2+resources_servers/swe_env(FakeSandbox: harnesses, grading, lifecycle, reaper, idempotency, egress, verifier wire-contract, decoupled cutover, mask-rejoin, infra-error row, flat-eval parser).pytest-dev__pytest-7982), the decoupled verifier and the official SWE-bench harness both resolve it (MATCH); alsoastropy__astropy-13453→ reward 1.0 on both docker and apptainer providers; a 500-instance SWE-bench Verified gold run graded via the decoupled provider matched official resolution.NemoGymClient/CodeActAgentdoesn't translate these locally-served models' tool-calls into actions — a fork↔model-integration detail orthogonal to this cutover, handled in production by the Gym model server + tuned models.)Deferred / infra-gated
.sifequivalence + opensandbox validation (need published.sifimages / a k8s cluster).data/example_rollouts.jsonlfrom a realng_collect_rolloutsrun beforeverified: true(current rows are synthetic placeholders).mainafter Add sandbox API and mini swe agent 2 resource agent NVIDIA-NeMo/Gym#1377 merges.🤖 Generated with Claude Code