Skip to content

feat(swe_env): decoupled SWE environment library + verifier (#1249) [1/3] #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adil-a wants to merge 1 commit into from
Closed

feat(swe_env): decoupled SWE environment library + verifier (#1249) [1/3] #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace).
| Swe Agents | | - | - | ✓ | ✓ | Apache 2.0 | <a href='responses_api_agents/swe_agents/configs/swebench_openhands.yaml'>swebench_openhands.yaml</a> | - |
| Swe Agents | | - | - | ✓ | ✓ | Apache 2.0 | <a href='responses_api_agents/swe_agents/configs/swebench_openhands_training.yaml'>swebench_openhands_training.yaml</a> | - |
| Swe Agents | coding | Software engineering tasks with OpenHands agent harness. | Improve agentic software engineering capabilities. | ✓ | ✓ | MIT | <a href='responses_api_agents/swe_agents/configs/swebench_swe_agent.yaml'>swebench_swe_agent.yaml</a> | - |
| Swe Env | software_engineering | SWE environment verifier (fresh sandbox, provider-neutral; decouples | - | - | - | - | <a href='resources_servers/swe_env/configs/swe_env.yaml'>swe_env.yaml</a> | - |
| Swe Pivot | agent | SWE pivot verifier for PivotRL on coding agent trajectories | Improve coding agent fix-design decisions | ✓ | ✓ | Apache 2.0 | <a href='resources_servers/swe_pivot/configs/swe_pivot.yaml'>swe_pivot.yaml</a> | - |
| Swerl Gen | coding | Running sandboxed evaluation for SWE-style tasks (either patch generation or reproduction test generation) | Improve SWE capabilities useful for benchmarks like SWE-bench | ✓ | ✓ | Apache 2.0 | <a href='resources_servers/swerl_gen/configs/swerl_gen.yaml'>swerl_gen.yaml</a> | - |
| Swerl Llm Judge | coding | SWE-style multiple-choice LLM-judge tasks scored via <solution>...</solution> choice. | Improve SWE capabilities useful for benchmarks like SWE-bench | ✓ | ✓ | MIT | <a href='resources_servers/swerl_llm_judge/configs/swerl_llm_judge.yaml'>swerl_llm_judge.yaml</a> | - |
Expand Down
76 changes: 76 additions & 0 deletions resources_servers/swe_env/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
<!--
Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->

# `swe_env` verifier

The required, provider-neutral, **fresh-sandbox** verification entry point for the
decoupled SWE environment (issue #1249). `verify()` takes an agent's patch, grades
it in its **own fresh sandbox**, and returns a non-nullable `reward` (`1.0`/`0.0`,
masked infra failures = `reward=0.0` + `mask_sample`). It imports the reusable
[`responses_api_agents/swe_env`](../../responses_api_agents/swe_env) library
(harness recipes, parsing, sandbox providers, lifecycle) — so any agent can reuse
the same env over HTTP, or in-process via that library.

Sandbox providers (selected by config `sandbox_provider`):
- **`docker`** — runs the SWE-bench eval Docker images directly (no `.sif` needed).
- **`apptainer`** — runs `.sif` images (ports the legacy on-prem path).
- **`opensandbox`** — the #1377 k8s provider (flat families).

## Running the full SWE-bench Verified eval (gold-patch validation)

`scripts/run_swebench_verified.py` runs the **decoupled sandbox infra over SWE-bench
Verified**: for each instance it provisions the official SWE-bench Docker image
through the `swe_env` provider + lifecycle, applies the **gold** patch, runs the real
per-repo SWE-bench `eval_script`, and grades with the official `swebench` parser. A
gold run should resolve ~all instances and validates the provider/lifecycle at full scale.

### Setup
```bash
# extra deps for the driver (not needed by the server itself)
uv pip install swebench datasets
# docker (default provider) must be installed; for --provider apptainer, apptainer + uidmap too.
```
SWE-bench images are pulled automatically from Docker Hub (`swebench` namespace,
`sweb.eval.x86_64.<instance_id>` with `__`→`_1776_`). The full Verified set needs
**~120 GB+** of disk; the driver `docker rmi`s each image after grading to bound usage.
There are **no pre-built `.sif` files**; `--provider apptainer` converts each image on
the fly (`apptainer build docker-daemon://…`). Set `HF_HOME` to a writable dir if your
`~/.cache/huggingface` is not writable.

### Examples
```bash
# smoke: first 5 instances on docker
python resources_servers/swe_env/scripts/run_swebench_verified.py --limit 5

# FULL 500, 4 in parallel, incremental results (resumable log)
python resources_servers/swe_env/scripts/run_swebench_verified.py \
--concurrency 4 --output results/swebench_verified_gold.jsonl

# apptainer provider (builds a .sif per instance, then removes it)
python resources_servers/swe_env/scripts/run_swebench_verified.py --provider apptainer --limit 5

# specific instances
python resources_servers/swe_env/scripts/run_swebench_verified.py \
--instances astropy__astropy-13453,django__django-11099
```
Flags: `--limit N`, `--instances id1,id2`, `--provider docker|apptainer`,
`--concurrency K`, `--eval-timeout S`, `--keep-images`, `--output PATH`.

Output: a per-instance line (`PASS`/`fail`/`ERR`) and a final
`resolved N/total (P%)`. Each result row (`{instance_id, resolved, status, error}`)
is appended to `--output` as it completes.

### Validated
- A real instance (`astropy__astropy-13453`) resolves end-to-end on **both** the
`docker` and `apptainer` providers (`reward=1.0`).
- Unit tests (FakeSandbox) + env-gated real-container tests live in `tests/`.

## Tests
```bash
RAY_TMPDIR=/tmp ng_test +entrypoint=resources_servers/swe_env
# env-gated real-container tests (need docker / apptainer):
SWE_ENV_DOCKER_ITEST=1 pytest resources_servers/swe_env/tests/test_verify.py -k docker_real
SWE_ENV_REAL_SWEBENCH=1 pytest resources_servers/swe_env/tests/test_swebench_real_instance.py
```
20 changes: 20 additions & 0 deletions resources_servers/swe_env/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""SWE environment verifier resources server package.

Provides the FastAPI resources server that grades an agent's patch for a
software-engineering task in a fresh, stateless sandbox and returns the reward
along with the eval-side outcome fields.
"""
Loading
Loading