diff --git a/README.md b/README.md
index 71d6e511af..26359217c3 100644
--- a/README.md
+++ b/README.md
@@ -286,6 +286,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace).
| Swe Agents | | - | - | ✓ | ✓ | Apache 2.0 | swebench_openhands.yaml | - |
| Swe Agents | | - | - | ✓ | ✓ | Apache 2.0 | swebench_openhands_training.yaml | - |
| Swe Agents | coding | Software engineering tasks with OpenHands agent harness. | Improve agentic software engineering capabilities. | ✓ | ✓ | MIT | swebench_swe_agent.yaml | - |
+| Swe Env | software_engineering | SWE environment verifier (fresh sandbox, provider-neutral; decouples | - | - | - | - | swe_env.yaml | - |
| Swe Pivot | agent | SWE pivot verifier for PivotRL on coding agent trajectories | Improve coding agent fix-design decisions | ✓ | ✓ | Apache 2.0 | swe_pivot.yaml | - |
| Swerl Gen | coding | Running sandboxed evaluation for SWE-style tasks (either patch generation or reproduction test generation) | Improve SWE capabilities useful for benchmarks like SWE-bench | ✓ | ✓ | Apache 2.0 | swerl_gen.yaml | - |
| Swerl Llm Judge | coding | SWE-style multiple-choice LLM-judge tasks scored via ... choice. | Improve SWE capabilities useful for benchmarks like SWE-bench | ✓ | ✓ | MIT | swerl_llm_judge.yaml | - |
diff --git a/resources_servers/swe_env/README.md b/resources_servers/swe_env/README.md
new file mode 100644
index 0000000000..a23727c97e
--- /dev/null
+++ b/resources_servers/swe_env/README.md
@@ -0,0 +1,76 @@
+
+
+# `swe_env` verifier
+
+The required, provider-neutral, **fresh-sandbox** verification entry point for the
+decoupled SWE environment (issue #1249). `verify()` takes an agent's patch, grades
+it in its **own fresh sandbox**, and returns a non-nullable `reward` (`1.0`/`0.0`,
+masked infra failures = `reward=0.0` + `mask_sample`). It imports the reusable
+[`responses_api_agents/swe_env`](../../responses_api_agents/swe_env) library
+(harness recipes, parsing, sandbox providers, lifecycle) — so any agent can reuse
+the same env over HTTP, or in-process via that library.
+
+Sandbox providers (selected by config `sandbox_provider`):
+- **`docker`** — runs the SWE-bench eval Docker images directly (no `.sif` needed).
+- **`apptainer`** — runs `.sif` images (ports the legacy on-prem path).
+- **`opensandbox`** — the #1377 k8s provider (flat families).
+
+## Running the full SWE-bench Verified eval (gold-patch validation)
+
+`scripts/run_swebench_verified.py` runs the **decoupled sandbox infra over SWE-bench
+Verified**: for each instance it provisions the official SWE-bench Docker image
+through the `swe_env` provider + lifecycle, applies the **gold** patch, runs the real
+per-repo SWE-bench `eval_script`, and grades with the official `swebench` parser. A
+gold run should resolve ~all instances and validates the provider/lifecycle at full scale.
+
+### Setup
+```bash
+# extra deps for the driver (not needed by the server itself)
+uv pip install swebench datasets
+# docker (default provider) must be installed; for --provider apptainer, apptainer + uidmap too.
+```
+SWE-bench images are pulled automatically from Docker Hub (`swebench` namespace,
+`sweb.eval.x86_64.` with `__`→`_1776_`). The full Verified set needs
+**~120 GB+** of disk; the driver `docker rmi`s each image after grading to bound usage.
+There are **no pre-built `.sif` files**; `--provider apptainer` converts each image on
+the fly (`apptainer build docker-daemon://…`). Set `HF_HOME` to a writable dir if your
+`~/.cache/huggingface` is not writable.
+
+### Examples
+```bash
+# smoke: first 5 instances on docker
+python resources_servers/swe_env/scripts/run_swebench_verified.py --limit 5
+
+# FULL 500, 4 in parallel, incremental results (resumable log)
+python resources_servers/swe_env/scripts/run_swebench_verified.py \
+ --concurrency 4 --output results/swebench_verified_gold.jsonl
+
+# apptainer provider (builds a .sif per instance, then removes it)
+python resources_servers/swe_env/scripts/run_swebench_verified.py --provider apptainer --limit 5
+
+# specific instances
+python resources_servers/swe_env/scripts/run_swebench_verified.py \
+ --instances astropy__astropy-13453,django__django-11099
+```
+Flags: `--limit N`, `--instances id1,id2`, `--provider docker|apptainer`,
+`--concurrency K`, `--eval-timeout S`, `--keep-images`, `--output PATH`.
+
+Output: a per-instance line (`PASS`/`fail`/`ERR`) and a final
+`resolved N/total (P%)`. Each result row (`{instance_id, resolved, status, error}`)
+is appended to `--output` as it completes.
+
+### Validated
+- A real instance (`astropy__astropy-13453`) resolves end-to-end on **both** the
+ `docker` and `apptainer` providers (`reward=1.0`).
+- Unit tests (FakeSandbox) + env-gated real-container tests live in `tests/`.
+
+## Tests
+```bash
+RAY_TMPDIR=/tmp ng_test +entrypoint=resources_servers/swe_env
+# env-gated real-container tests (need docker / apptainer):
+SWE_ENV_DOCKER_ITEST=1 pytest resources_servers/swe_env/tests/test_verify.py -k docker_real
+SWE_ENV_REAL_SWEBENCH=1 pytest resources_servers/swe_env/tests/test_swebench_real_instance.py
+```
diff --git a/resources_servers/swe_env/__init__.py b/resources_servers/swe_env/__init__.py
new file mode 100644
index 0000000000..435f4d3534
--- /dev/null
+++ b/resources_servers/swe_env/__init__.py
@@ -0,0 +1,20 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""SWE environment verifier resources server package.
+
+Provides the FastAPI resources server that grades an agent's patch for a
+software-engineering task in a fresh, stateless sandbox and returns the reward
+along with the eval-side outcome fields.
+"""
diff --git a/resources_servers/swe_env/app.py b/resources_servers/swe_env/app.py
new file mode 100644
index 0000000000..4df68f692c
--- /dev/null
+++ b/resources_servers/swe_env/app.py
@@ -0,0 +1,250 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""SWE environment verifier resources server.
+
+A ``SimpleResourcesServer`` whose ``verify()`` extracts the agent's patch from
+the response, builds a ``SweTask`` from the per-task metadata, grades it in its
+own fresh, stateless sandbox via the ``verify_task`` orchestrator, and returns
+the eval-side fields plus reward.
+
+For the apptainer provider it must co-locate with the ``.sif``/Lustre storage
+(and Docker for nested families). The reward is a non-nullable float; masking is
+carried as ``reward=0.0`` plus ``mask_sample``.
+"""
+
+from __future__ import annotations
+
+import json
+import re
+from typing import Any
+
+from nemo_gym.base_resources_server import (
+ BaseResourcesServerConfig,
+ BaseVerifyRequest,
+ BaseVerifyResponse,
+ SimpleResourcesServer,
+)
+from nemo_gym.openai_utils import NeMoGymResponse
+from resources_servers.swe_env.verify_task import verify_task
+from responses_api_agents.swe_env.grading import reward_from_report
+from responses_api_agents.swe_env.harness import SweTask
+
+
+_FENCED_DIFF = re.compile(r"```(?:diff|patch)?\s*\n(.*?)```", re.DOTALL)
+
+
+def _as_list(value: Any) -> list[str]:
+ """Coerce a metadata value into a list of strings.
+
+ Accepts ``None`` (yields an empty list), an existing list (each element
+ stringified), or a string. A string that looks like a JSON array is parsed
+ into its elements; any other non-empty string becomes a single-element list.
+
+ Args:
+ value: The raw metadata value to normalize.
+
+ Returns:
+ list[str]: The value expressed as a list of strings.
+ """
+ if value is None:
+ return []
+ if isinstance(value, list):
+ return [str(v) for v in value]
+ if isinstance(value, str):
+ stripped = value.strip()
+ if stripped.startswith("["):
+ try:
+ return [str(v) for v in json.loads(stripped)]
+ except json.JSONDecodeError:
+ pass
+ return [stripped] if stripped else []
+ return [str(value)]
+
+
+class SweEnvVerifierConfig(BaseResourcesServerConfig):
+ """Configuration for the SWE environment verifier.
+
+ Attributes:
+ sandbox_provider: Single-key provider mapping selecting the sandbox
+ backend (e.g. ``{"docker": {}}``).
+ model_patch_field: Metadata key under which the agent's patch is stored.
+ opensandbox_service_url: Optional URL of an opensandbox service.
+ """
+
+ sandbox_provider: dict[str, Any] = {"docker": {}}
+ model_patch_field: str = "model_patch"
+ opensandbox_service_url: str | None = None
+
+
+class SweEnvVerifyResponse(BaseVerifyResponse):
+ """Verify response carrying the SWE eval outcome alongside the reward.
+
+ Attributes:
+ resolved: Whether the patch resolved the task.
+ patch_exists: Whether a non-empty patch was supplied.
+ patch_applied: Whether the patch applied cleanly in the sandbox.
+ eval_error: Whether evaluation failed and the sample is masked.
+ error_kind: Typed error category when evaluation failed, else ``None``.
+ mask_sample: Whether the sample should be excluded from training.
+ instance_id: Identifier of the graded instance.
+ """
+
+ resolved: bool = False
+ patch_exists: bool = False
+ patch_applied: bool = False
+ eval_error: bool = False
+ error_kind: str | None = None
+ mask_sample: bool = False
+ instance_id: str = ""
+
+
+class SweEnvVerifier(SimpleResourcesServer):
+ """Resources server that grades an agent's patch in a fresh sandbox."""
+
+ config: SweEnvVerifierConfig
+
+ async def verify(self, body: BaseVerifyRequest) -> SweEnvVerifyResponse:
+ """Grade the agent's patch for one task and build the verify response.
+
+ Builds a ``SweTask`` from the request, evaluates it in a fresh sandbox,
+ converts the eval report into a reward, and masks the sample when the
+ report carries an error.
+
+ Args:
+ body: The verify request with per-task metadata and agent response.
+
+ Returns:
+ SweEnvVerifyResponse: The reward together with the eval-side fields.
+ """
+ task = build_task(body, self.config.model_patch_field)
+ report = await verify_task(self.config.sandbox_provider, task)
+ reward = reward_from_report(report)
+ masked = report.error_kind is not None
+ return SweEnvVerifyResponse(
+ **body.model_dump(),
+ reward=reward,
+ resolved=report.resolved,
+ patch_exists=report.patch_exists,
+ patch_applied=report.patch_applied,
+ eval_error=masked,
+ error_kind=report.error_kind,
+ mask_sample=masked,
+ instance_id=report.instance_id,
+ )
+
+
+def build_task(body: BaseVerifyRequest, patch_field: str) -> SweTask:
+ """Map a verify request onto a SweTask.
+
+ Reads the per-task metadata and the agent response and assembles the task
+ fields the eval harness needs. Module-level (not a method) so it is
+ unit-testable without instantiating the Pydantic server.
+
+ Args:
+ body: The verify request with per-task metadata and agent response.
+ patch_field: Metadata key under which the agent's patch is stored.
+
+ Returns:
+ SweTask: The task to grade, including the extracted model patch.
+ """
+ metadata: dict[str, Any] = dict(body.responses_create_params.metadata or {})
+ # Some rows nest task fields inside a stringified ``instance_dict`` (e.g.
+ # fail_to_pass_select / base_dockerfile). Surface those keys at top level so the
+ # ``*_select`` + dockerfile-ENV handling and the other harnesses find them;
+ # explicit top-level metadata keys take precedence.
+ instance_dict = metadata.get("instance_dict")
+ if isinstance(instance_dict, str):
+ try:
+ instance_dict = json.loads(instance_dict)
+ except json.JSONDecodeError:
+ instance_dict = None
+ if isinstance(instance_dict, dict):
+ metadata = {**instance_dict, **metadata}
+ patch = extract_patch(body.response, metadata, patch_field)
+ return SweTask(
+ instance_id=str(metadata.get("instance_id", "unknown")),
+ image=metadata.get("image"),
+ base_commit=metadata.get("base_commit"),
+ repo_workdir=str(metadata.get("repo_workdir", "/testbed")),
+ test_command=str(metadata.get("test_command", "")),
+ test_framework=str(metadata.get("test_framework", "")),
+ model_patch=patch,
+ test_patch=str(metadata.get("test_patch", "")),
+ # Some rows carry the required-test lists under UPPERCASE FAIL_TO_PASS /
+ # PASS_TO_PASS keys. Reading lowercase only would leave those rows with empty
+ # required-test lists, which the rebench/ext resolution rule (empty subset <= any
+ # set) scores resolved=True for EVERY sample -> reward inflation. Fall back to the
+ # UPPERCASE keys.
+ fail_to_pass=_as_list(metadata.get("fail_to_pass") or metadata.get("FAIL_TO_PASS")),
+ pass_to_pass=_as_list(metadata.get("pass_to_pass") or metadata.get("PASS_TO_PASS")),
+ benchmark=str(metadata.get("benchmark", "swe-bench-ext")),
+ split=str(metadata.get("split", "test")),
+ metadata=metadata,
+ )
+
+
+def extract_patch(response: NeMoGymResponse, metadata: dict[str, Any], patch_field: str) -> str:
+ """Read the patch from the response.
+
+ Prefers the normalized patch stored on the response metadata; if absent,
+ falls back to the first fenced ``diff``/``patch`` block found in the output
+ text.
+
+ Args:
+ response: The agent response to read the patch from.
+ metadata: Per-task metadata (unused for lookup but kept for parity).
+ patch_field: Metadata key under which the patch is stored.
+
+ Returns:
+ str: The extracted patch, or an empty string if none is found.
+ """
+ response_metadata = getattr(response, "metadata", None) or {}
+ patch = response_metadata.get(patch_field)
+ if patch:
+ return str(patch)
+ for item in getattr(response, "output", []) or []:
+ text = _item_text(item)
+ if text:
+ match = _FENCED_DIFF.search(text)
+ if match:
+ return match.group(1)
+ return ""
+
+
+def _item_text(item: Any) -> str:
+ """Extract the text content from a response output item.
+
+ Args:
+ item: A response output item whose ``content`` may be a string or a list
+ of content chunks each carrying a ``text`` attribute.
+
+ Returns:
+ str: The concatenated text, or an empty string if there is none.
+ """
+ content = getattr(item, "content", None)
+ if isinstance(content, str):
+ return content
+ if isinstance(content, list):
+ parts = []
+ for chunk in content:
+ text = getattr(chunk, "text", None)
+ if text:
+ parts.append(text)
+ return "\n".join(parts)
+ return ""
+
+
+if __name__ == "__main__":
+ SweEnvVerifier.run_webserver()
diff --git a/resources_servers/swe_env/configs/swe_env.yaml b/resources_servers/swe_env/configs/swe_env.yaml
new file mode 100644
index 0000000000..6bd5f59754
--- /dev/null
+++ b/resources_servers/swe_env/configs/swe_env.yaml
@@ -0,0 +1,33 @@
+# SWE environment verifier (#1249) — the required, fresh-sandbox, provider-neutral
+# verification entry point. A config "trio": the verifier resources server, an
+# agent that points at it, and a 5-row swe-bench-ext example dataset.
+#
+# NOTE: the committed data/example_rollouts.jsonl are synthetic gold-patch-injecting
+# placeholders so the ng_test_all data gate passes; regenerate them from a real
+# `ng_collect_rollouts` run before flipping `verified: true` (see SWE_ENV_DECOUPLE_STATUS.md).
+swe_env_resources_server:
+ resources_servers:
+ swe_env:
+ entrypoint: app.py
+ domain: software_engineering
+ verified: false
+ description: SWE environment verifier (fresh sandbox, provider-neutral; decouples #1249)
+ # Single-key provider mapping. 'docker' runs locally; 'apptainer' for on-prem .sif;
+ # 'opensandbox' for the #1377 k8s provider (flat families only).
+ sandbox_provider:
+ docker: {}
+
+swe_env_simple_agent:
+ responses_api_agents:
+ simple_agent:
+ entrypoint: app.py
+ resources_server:
+ type: resources_servers
+ name: swe_env_resources_server
+ model_server:
+ type: responses_api_models
+ name: policy_model
+ datasets:
+ - name: example
+ type: example
+ jsonl_fpath: resources_servers/swe_env/data/example.jsonl
diff --git a/resources_servers/swe_env/data/example.jsonl b/resources_servers/swe_env/data/example.jsonl
new file mode 100644
index 0000000000..067492cb18
--- /dev/null
+++ b/resources_servers/swe_env/data/example.jsonl
@@ -0,0 +1,5 @@
+{"id": 0, "agent_ref": {"name": "swe_env_patch_injecting_agent"}, "responses_create_params": {"input": [{"role": "user", "content": "Fix the bug in instance 0 (calc.add subtracts instead of adds)."}], "metadata": {"instance_id": "swe-bench-ext-example-0", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "verifier_metadata": {"instance_id": "swe-bench-ext-example-0", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}
+{"id": 1, "agent_ref": {"name": "swe_env_patch_injecting_agent"}, "responses_create_params": {"input": [{"role": "user", "content": "Fix the bug in instance 1 (calc.add subtracts instead of adds)."}], "metadata": {"instance_id": "swe-bench-ext-example-1", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "verifier_metadata": {"instance_id": "swe-bench-ext-example-1", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}
+{"id": 2, "agent_ref": {"name": "swe_env_patch_injecting_agent"}, "responses_create_params": {"input": [{"role": "user", "content": "Fix the bug in instance 2 (calc.add subtracts instead of adds)."}], "metadata": {"instance_id": "swe-bench-ext-example-2", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "verifier_metadata": {"instance_id": "swe-bench-ext-example-2", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}
+{"id": 3, "agent_ref": {"name": "swe_env_patch_injecting_agent"}, "responses_create_params": {"input": [{"role": "user", "content": "Fix the bug in instance 3 (calc.add subtracts instead of adds)."}], "metadata": {"instance_id": "swe-bench-ext-example-3", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "verifier_metadata": {"instance_id": "swe-bench-ext-example-3", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}
+{"id": 4, "agent_ref": {"name": "swe_env_patch_injecting_agent"}, "responses_create_params": {"input": [{"role": "user", "content": "Fix the bug in instance 4 (calc.add subtracts instead of adds)."}], "metadata": {"instance_id": "swe-bench-ext-example-4", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "verifier_metadata": {"instance_id": "swe-bench-ext-example-4", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}
diff --git a/resources_servers/swe_env/data/example_metrics.json b/resources_servers/swe_env/data/example_metrics.json
new file mode 100644
index 0000000000..a8c1702c12
--- /dev/null
+++ b/resources_servers/swe_env/data/example_metrics.json
@@ -0,0 +1,17 @@
+{
+ "name": "example",
+ "type": "example",
+ "jsonl_fpath": "resources_servers/swe_env/data/example.jsonl",
+ "num_repeats": 1,
+ "gitlab_identifier": null,
+ "huggingface_identifier": null,
+ "license": null,
+ "Number of examples": 5,
+ "Number of turns": {
+ "Total # non-null values": 5,
+ "Average": 1.0,
+ "Min": 1.0,
+ "Max": 1.0,
+ "Standard deviation": 0.0
+ }
+}
\ No newline at end of file
diff --git a/resources_servers/swe_env/data/example_rollouts.jsonl b/resources_servers/swe_env/data/example_rollouts.jsonl
new file mode 100644
index 0000000000..a22c772ce1
--- /dev/null
+++ b/resources_servers/swe_env/data/example_rollouts.jsonl
@@ -0,0 +1,5 @@
+{"id": 0, "responses_create_params": {"input": [{"role": "user", "content": "Fix the bug in instance 0."}], "metadata": {"instance_id": "swe-bench-ext-example-0", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "response": {"id": "swebench-ext-example-0", "object": "response", "output": [], "metadata": {"model_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "reward": 1.0, "resolved": true, "patch_exists": true, "patch_applied": true, "mask_sample": false, "instance_id": "swe-bench-ext-example-0"}
+{"id": 1, "responses_create_params": {"input": [{"role": "user", "content": "Fix the bug in instance 1."}], "metadata": {"instance_id": "swe-bench-ext-example-1", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "response": {"id": "swebench-ext-example-1", "object": "response", "output": [], "metadata": {"model_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "reward": 1.0, "resolved": true, "patch_exists": true, "patch_applied": true, "mask_sample": false, "instance_id": "swe-bench-ext-example-1"}
+{"id": 2, "responses_create_params": {"input": [{"role": "user", "content": "Fix the bug in instance 2."}], "metadata": {"instance_id": "swe-bench-ext-example-2", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "response": {"id": "swebench-ext-example-2", "object": "response", "output": [], "metadata": {"model_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "reward": 1.0, "resolved": true, "patch_exists": true, "patch_applied": true, "mask_sample": false, "instance_id": "swe-bench-ext-example-2"}
+{"id": 3, "responses_create_params": {"input": [{"role": "user", "content": "Fix the bug in instance 3."}], "metadata": {"instance_id": "swe-bench-ext-example-3", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "response": {"id": "swebench-ext-example-3", "object": "response", "output": [], "metadata": {"model_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "reward": 1.0, "resolved": true, "patch_exists": true, "patch_applied": true, "mask_sample": false, "instance_id": "swe-bench-ext-example-3"}
+{"id": 4, "responses_create_params": {"input": [{"role": "user", "content": "Fix the bug in instance 4."}], "metadata": {"instance_id": "swe-bench-ext-example-4", "image": "swe-env-itest:local", "base_commit": "HEAD", "repo_workdir": "/testbed", "test_command": "python -m pytest -rA -q", "test_framework": "pytest", "fail_to_pass": "[\"test_calc.py::test_add\"]", "benchmark": "swe-bench-ext", "split": "test", "golden_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "response": {"id": "swebench-ext-example-4", "object": "response", "output": [], "metadata": {"model_patch": "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"}}, "reward": 1.0, "resolved": true, "patch_exists": true, "patch_applied": true, "mask_sample": false, "instance_id": "swe-bench-ext-example-4"}
diff --git a/resources_servers/swe_env/requirements.txt b/resources_servers/swe_env/requirements.txt
new file mode 100644
index 0000000000..00ed83213e
--- /dev/null
+++ b/resources_servers/swe_env/requirements.txt
@@ -0,0 +1 @@
+-e nemo-gym[dev] @ ../../
diff --git a/resources_servers/swe_env/scripts/run_swebench_verified.py b/resources_servers/swe_env/scripts/run_swebench_verified.py
new file mode 100644
index 0000000000..63b35669ae
--- /dev/null
+++ b/resources_servers/swe_env/scripts/run_swebench_verified.py
@@ -0,0 +1,252 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Run the swe_env sandbox infra over SWE-bench Verified (gold-patch eval).
+
+For each instance: provision the official SWE-bench docker image through the
+swe_env sandbox provider and lifecycle (``acquire_sandbox``), apply the gold
+patch, run the SWE-bench ``eval_script`` (the real per-repo test command), and
+grade with the official ``swebench`` parser. A gold run should resolve nearly
+all instances.
+
+This is a driver/operational script (not a unit test). Requires extra deps:
+ uv pip install swebench datasets # + docker (provider=docker) or apptainer
+
+Examples:
+ # smoke (5 instances), docker provider, prune images to bound disk
+ python resources_servers/swe_env/scripts/run_swebench_verified.py --limit 5
+
+ # full 500, 4 in parallel, keep an incremental results file
+ python resources_servers/swe_env/scripts/run_swebench_verified.py \\
+ --concurrency 4 --output /tmp/swebench_gold_results.jsonl
+
+ # apptainer provider (converts each image to .sif on the fly, then removes it)
+ python resources_servers/swe_env/scripts/run_swebench_verified.py --provider apptainer --limit 5
+"""
+
+from __future__ import annotations
+
+import argparse
+import asyncio
+import json
+import os
+import subprocess
+import sys
+import tempfile
+from pathlib import Path
+
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
+# Use a writable HF cache (the default ~/.cache/huggingface may be root-polluted by
+# docker containers that mount it). Override with HF_HOME in the environment.
+os.environ.setdefault("HF_HOME", str(Path(__file__).resolve().parents[3] / ".hf_cache"))
+
+import responses_api_agents.swe_env.providers # noqa: E402,F401 (registers docker + apptainer providers)
+from nemo_gym.sandbox import SandboxSpec # noqa: E402
+from responses_api_agents.swe_env.lifecycle import acquire_sandbox # noqa: E402
+
+
+def _load_instances(limit, instance_ids):
+ """Load SWE-bench Verified test instances.
+
+ Args:
+ limit: Maximum number of instances to return, or a falsy value for all.
+ instance_ids: Optional iterable of instance ids to filter to.
+
+ Returns:
+ list[dict]: The selected dataset rows.
+ """
+ from datasets import load_dataset
+
+ ds = load_dataset("princeton-nlp/SWE-bench_Verified", split="test")
+ rows = list(ds)
+ if instance_ids:
+ wanted = set(instance_ids)
+ rows = [r for r in rows if r["instance_id"] in wanted]
+ if limit:
+ rows = rows[:limit]
+ return rows
+
+
+def _docker(*args, timeout=None):
+ """Run a ``docker`` subcommand, capturing its output.
+
+ Args:
+ *args: Arguments passed through to the ``docker`` CLI.
+ timeout: Optional timeout in seconds for the subprocess.
+
+ Returns:
+ subprocess.CompletedProcess: The completed process with captured output.
+ """
+ return subprocess.run(["docker", *args], capture_output=True, text=True, timeout=timeout)
+
+
+def _build_sif(image, sif_path):
+ """Build an apptainer ``.sif`` image from a local docker image.
+
+ Args:
+ image: The docker image reference to convert.
+ sif_path: Destination path for the built ``.sif`` file.
+ """
+ subprocess.run(
+ ["apptainer", "build", "--force", sif_path, f"docker-daemon://{image}"],
+ check=True,
+ capture_output=True,
+ )
+
+
+async def _eval_one(instance, *, provider_name, keep_images, eval_timeout):
+ """Evaluate one SWE-bench Verified instance with its gold patch.
+
+ Pulls the official image, provisions a sandbox through the selected
+ provider, applies the gold patch, runs the SWE-bench eval script, parses the
+ logs, and computes whether the instance resolved. Docker (and any built
+ ``.sif``) images are removed afterwards unless ``keep_images`` is set.
+
+ Args:
+ instance: The dataset row for the instance.
+ provider_name: Sandbox provider to use (``"docker"`` or ``"apptainer"``).
+ keep_images: When True, do not remove images after the run.
+ eval_timeout: Eval-script timeout in seconds.
+
+ Returns:
+ dict: Result with ``instance_id``, ``resolved``, ``status``, and
+ ``error`` keys.
+ """
+ from swebench.harness.constants import FAIL_TO_PASS, PASS_TO_PASS, TestStatus
+ from swebench.harness.grading import get_logs_eval
+ from swebench.harness.test_spec.test_spec import make_test_spec
+
+ iid = instance["instance_id"]
+ # namespace="swebench" -> Docker Hub image key (swebench/sweb.eval.x86_64.:latest);
+ # the default (None) yields a namespace-less local name that isn't pullable.
+ spec = make_test_spec(instance, namespace="swebench")
+ image = spec.instance_image_key
+ sif_path = None
+ try:
+ _docker("pull", image, timeout=3600)
+ if provider_name == "apptainer":
+ sif_path = f"/tmp/sweb-{iid}.sif"
+ _build_sif(image, sif_path)
+ provider = {"apptainer": {}}
+ sbox_image = iid
+ provider_options = {"sif_path": sif_path}
+ else:
+ provider = {"docker": {}}
+ sbox_image = image
+ provider_options = {}
+
+ sandbox_spec = SandboxSpec(
+ image=sbox_image,
+ workdir="/testbed",
+ ttl_s=eval_timeout + 600,
+ ready_timeout_s=900,
+ provider_options=provider_options,
+ )
+ async with acquire_sandbox(provider, sandbox_spec, instance_id=iid) as env:
+ await env.write_text("/root/gold.patch", instance["patch"])
+ await env.execute(
+ "cd /testbed && (git apply -v /root/gold.patch || git apply -v --3way /root/gold.patch)",
+ cwd="/testbed",
+ )
+ await env.write_text("/root/eval.sh", spec.eval_script)
+ result = await env.execute("bash /root/eval.sh", timeout_s=eval_timeout, is_eval=True)
+
+ with tempfile.NamedTemporaryFile("w", suffix=".log", delete=False) as fh:
+ fh.write(result.get("output", ""))
+ log_path = fh.name
+ # swebench's per-repo log parser -> {test_id: status}; we compute resolution
+ # ourselves from the instance's gold FAIL_TO_PASS / PASS_TO_PASS.
+ status_map, found = get_logs_eval(spec, log_path)
+ Path(log_path).unlink(missing_ok=True)
+ f2p = instance.get(FAIL_TO_PASS) or []
+ p2p = instance.get(PASS_TO_PASS) or []
+ if isinstance(f2p, str):
+ f2p = json.loads(f2p)
+ if isinstance(p2p, str):
+ p2p = json.loads(p2p)
+ passed = {t for t, s in status_map.items() if s == TestStatus.PASSED.value}
+ resolved = bool(found) and all(t in passed for t in f2p) and all(t in passed for t in p2p)
+ status = "RESOLVED" if resolved else ("NO_LOG" if not found else "UNRESOLVED")
+ return {"instance_id": iid, "resolved": resolved, "status": status, "error": None}
+ except Exception as exc: # noqa: BLE001
+ return {"instance_id": iid, "resolved": False, "status": "ERROR", "error": repr(exc)}
+ finally:
+ if not keep_images:
+ _docker("rmi", "-f", image)
+ if sif_path:
+ Path(sif_path).unlink(missing_ok=True)
+
+
+async def _main_async(args):
+ """Evaluate the selected instances concurrently and print a summary.
+
+ Loads the instances, runs them with bounded concurrency, optionally streams
+ each result to an output JSONL file, and prints per-instance progress plus a
+ final resolved/error summary.
+
+ Args:
+ args: Parsed command-line arguments.
+ """
+ instances = _load_instances(args.limit, args.instances.split(",") if args.instances else None)
+ print(f"Running {len(instances)} SWE-bench Verified instances (gold) via provider={args.provider}", flush=True)
+ sem = asyncio.Semaphore(args.concurrency)
+ out = open(args.output, "w") if args.output else None
+ results = []
+
+ async def _runner(inst):
+ """Evaluate one instance under the concurrency semaphore and record it.
+
+ Args:
+ inst: The dataset row for the instance to evaluate.
+ """
+ async with sem:
+ res = await _eval_one(
+ inst,
+ provider_name=args.provider,
+ keep_images=args.keep_images,
+ eval_timeout=args.eval_timeout,
+ )
+ results.append(res)
+ mark = "PASS" if res["resolved"] else ("ERR " if res["error"] else "fail")
+ print(f" [{len(results):>3}/{len(instances)}] {mark} {res['instance_id']} ({res['status']})", flush=True)
+ if out:
+ out.write(json.dumps(res) + "\n")
+ out.flush()
+
+ await asyncio.gather(*[_runner(i) for i in instances])
+ if out:
+ out.close()
+ resolved = sum(r["resolved"] for r in results)
+ errors = sum(1 for r in results if r["error"])
+ print(
+ f"\n=== RESULT: resolved {resolved}/{len(results)} ({100 * resolved / max(1, len(results)):.1f}%); errors {errors} ==="
+ )
+
+
+def main():
+ """Parse command-line arguments and run the gold-patch evaluation."""
+ p = argparse.ArgumentParser(description="Gold-patch eval of SWE-bench Verified via swe_env providers")
+ p.add_argument("--limit", type=int, default=None, help="only the first N instances")
+ p.add_argument("--instances", type=str, default="", help="comma-separated instance_ids")
+ p.add_argument("--provider", choices=["docker", "apptainer"], default="docker")
+ p.add_argument("--concurrency", type=int, default=2)
+ p.add_argument("--eval-timeout", type=int, default=1800)
+ p.add_argument("--keep-images", action="store_true", help="do not docker rmi after each instance")
+ p.add_argument("--output", type=str, default="", help="incremental results JSONL path")
+ asyncio.run(_main_async(p.parse_args()))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/resources_servers/swe_env/tests/__init__.py b/resources_servers/swe_env/tests/__init__.py
new file mode 100644
index 0000000000..bc89bd7262
--- /dev/null
+++ b/resources_servers/swe_env/tests/__init__.py
@@ -0,0 +1 @@
+"""Test package for the swe_env resources server."""
diff --git a/resources_servers/swe_env/tests/test_apptainer_itest.py b/resources_servers/swe_env/tests/test_apptainer_itest.py
new file mode 100644
index 0000000000..a9615e248a
--- /dev/null
+++ b/resources_servers/swe_env/tests/test_apptainer_itest.py
@@ -0,0 +1,68 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Real apptainer-provider end-to-end (env-gated; never runs in CI).
+
+Builds a ``.sif`` from the docker itest image (the calc-bug git repo) and runs
+``verify_task`` through the ApptainerSandboxProvider on the flat swe-bench-ext
+path. Enable with ``SWE_ENV_APPTAINER_ITEST=1`` on a box with apptainer + docker.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import os
+import shutil
+import subprocess
+import sys
+
+import pytest
+
+import responses_api_agents.swe_env.harnesses # noqa: F401 (register harnesses)
+from resources_servers.swe_env.verify_task import verify_task
+from responses_api_agents.swe_env.grading import reward_from_report
+from responses_api_agents.swe_env.harness import SweTask
+
+
+_RUN = os.environ.get("SWE_ENV_APPTAINER_ITEST") == "1" and shutil.which("apptainer") is not None
+_SIF = "/tmp/swe-env-itest.sif"
+_GOLD = "--- a/calc.py\n+++ b/calc.py\n@@ -1,2 +1,2 @@\n def add(a, b):\n- return a - b\n+ return a + b\n"
+
+
+@pytest.mark.skipif(not _RUN, reason="set SWE_ENV_APPTAINER_ITEST=1 and install apptainer")
+def test_apptainer_real_end_to_end():
+ """Build the .sif if absent, then verify the gold patch resolves via the apptainer provider."""
+ if not os.path.exists(_SIF):
+ build = subprocess.run(
+ ["apptainer", "build", "--force", _SIF, "docker-daemon://swe-env-itest:local"],
+ capture_output=True,
+ )
+ assert build.returncode == 0, build.stderr.decode(errors="replace")[-3000:]
+
+ task = SweTask(
+ instance_id="calc-apptainer",
+ image="swe-env-itest",
+ base_commit="HEAD",
+ repo_workdir="/testbed",
+ test_command="python -m pytest -rA -q",
+ model_patch=_GOLD,
+ fail_to_pass=["test_calc.py::test_add"],
+ benchmark="swe-bench-ext",
+ metadata={"provider_options": {"sif_path": _SIF}},
+ )
+ report = asyncio.run(verify_task({"apptainer": {}}, task))
+ sys.stderr.write(f"\n[apptainer itest] {report}\n")
+ assert report.patch_applied is True
+ assert report.resolved is True
+ assert reward_from_report(report) == 1.0
diff --git a/resources_servers/swe_env/tests/test_swebench_real_instance.py b/resources_servers/swe_env/tests/test_swebench_real_instance.py
new file mode 100644
index 0000000000..d58c4854d3
--- /dev/null
+++ b/resources_servers/swe_env/tests/test_swebench_real_instance.py
@@ -0,0 +1,111 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Real SWE-bench Verified instance end-to-end (env-gated; never runs in CI).
+
+Pulls the public SWE-bench docker image for one instance and grades its GOLD
+patch through the verifier (docker provider) — proving the verifier works on
+REAL benchmark data, not synthetic. Validated locally on both providers:
+``astropy__astropy-13453`` -> resolved=True, reward=1.0 (docker AND the
+docker->.sif apptainer path; build the .sif with
+``apptainer build x.sif docker-daemon://swebench/sweb.eval.x86_64.``).
+
+Enable with ``SWE_ENV_REAL_SWEBENCH=1`` (needs docker + network; pulls ~2.7GB).
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+import shutil
+import urllib.request
+
+import pytest
+
+from resources_servers.swe_env.verify_task import verify_task
+from responses_api_agents.swe_env.grading import reward_from_report
+from responses_api_agents.swe_env.harness import SweTask
+
+
+_RUN = os.environ.get("SWE_ENV_REAL_SWEBENCH") == "1" and shutil.which("docker") is not None
+_INSTANCE = "astropy__astropy-13453"
+_OFFSET = 4 # index of _INSTANCE in SWE-bench_Verified test split
+_DATASET_URL = (
+ "https://datasets-server.huggingface.co/rows"
+ f"?dataset=princeton-nlp/SWE-bench_Verified&config=default&split=test&offset={_OFFSET}&length=1"
+)
+
+
+def _image_for(instance_id: str) -> str:
+ """Map a SWE-bench instance id to its public Docker Hub image name.
+
+ Args:
+ instance_id: The SWE-bench instance id (``__`` separator).
+
+ Returns:
+ str: The Docker Hub image name for that instance.
+ """
+ # SWE-bench Docker Hub naming: __ -> _1776_, lowercased.
+ return "swebench/sweb.eval.x86_64." + instance_id.replace("__", "_1776_").lower()
+
+
+def _as_list(value):
+ """Normalize a value into a list, parsing JSON strings when possible.
+
+ Args:
+ value: A list, a JSON-encoded string, a bare string, or None.
+
+ Returns:
+ list: The parsed/normalized list (empty for None).
+ """
+ if isinstance(value, str):
+ try:
+ return json.loads(value)
+ except json.JSONDecodeError:
+ return [value]
+ return value or []
+
+
+@pytest.mark.skipif(not _RUN, reason="set SWE_ENV_REAL_SWEBENCH=1 (needs docker + network, pulls ~2.7GB)")
+def test_real_swebench_gold_patch_resolves():
+ """Pull a real SWE-bench Verified instance and verify its gold patch resolves to reward 1.0."""
+ with urllib.request.urlopen(_DATASET_URL, timeout=60) as resp:
+ row = json.load(resp)["rows"][0]["row"]
+ assert row["instance_id"] == _INSTANCE
+
+ f2p = _as_list(row.get("FAIL_TO_PASS"))
+ p2p = _as_list(row.get("PASS_TO_PASS"))
+ nodeids = " ".join("'" + n + "'" for n in f2p + p2p)
+ test_command = (
+ f"source /opt/miniconda3/etc/profile.d/conda.sh && conda activate testbed && python -m pytest -rA {nodeids}"
+ )
+ task = SweTask(
+ instance_id=_INSTANCE,
+ image=_image_for(_INSTANCE),
+ base_commit=row["base_commit"],
+ repo_workdir="/testbed",
+ test_command=test_command,
+ model_patch=row["patch"],
+ test_patch=row.get("test_patch", ""),
+ fail_to_pass=f2p,
+ pass_to_pass=p2p,
+ benchmark="swe-bench-ext",
+ metadata={"ttl_s": 3600, "ready_timeout_s": 900},
+ )
+
+ report = asyncio.run(verify_task({"docker": {}}, task))
+ assert report.patch_applied is True
+ assert report.resolved is True
+ assert reward_from_report(report) == 1.0
diff --git a/resources_servers/swe_env/tests/test_verify.py b/resources_servers/swe_env/tests/test_verify.py
new file mode 100644
index 0000000000..143e66aaa8
--- /dev/null
+++ b/resources_servers/swe_env/tests/test_verify.py
@@ -0,0 +1,347 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Verifier tests: the verify() adapter logic, reward correctness (FakeSandbox),
+and a real docker-backed end-to-end (env-gated so CI never runs it)."""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+import shutil
+import subprocess
+import sys
+from types import SimpleNamespace
+
+import pytest
+
+import responses_api_agents.swe_env.harnesses # noqa: F401 (register harnesses)
+from nemo_gym.sandbox import SandboxExecResult, SandboxHandle, SandboxStatus, register_provider
+from resources_servers.swe_env.app import _as_list, _item_text, build_task, extract_patch
+from resources_servers.swe_env.verify_task import verify_task
+from responses_api_agents.swe_env.grading import reward_from_report
+from responses_api_agents.swe_env.harness import SweTask
+
+
+# ----- verify() adapter logic (no HTTP / no pydantic construction needed) -----
+
+
+def test_as_list():
+ """Check that _as_list normalizes None, lists, JSON strings, and bare strings."""
+ assert _as_list(None) == []
+ assert _as_list(["a", "b"]) == ["a", "b"]
+ assert _as_list('["x", "y"]') == ["x", "y"]
+ assert _as_list("single") == ["single"]
+
+
+def test_extract_patch_from_metadata_field():
+ """Check that extract_patch reads the patch from the named response metadata field."""
+ response = SimpleNamespace(metadata={"model_patch": "diff --git a/x b/x\n"}, output=[])
+ assert extract_patch(response, {}, "model_patch") == "diff --git a/x b/x\n"
+
+
+def test_extract_patch_from_fenced_diff():
+ """Check that extract_patch recovers a diff from a fenced ```diff block in output."""
+ item = SimpleNamespace(content="here:\n```diff\n--- a/x\n+++ b/x\n```\n")
+ response = SimpleNamespace(metadata={}, output=[item])
+ assert "--- a/x" in extract_patch(response, {}, "model_patch")
+
+
+def test_task_from_request_maps_metadata():
+ """Check that build_task maps request metadata and the response patch onto a SweTask."""
+ body = SimpleNamespace(
+ responses_create_params=SimpleNamespace(
+ metadata={
+ "instance_id": "abc",
+ "image": "img:tag",
+ "base_commit": "deadbeef",
+ "test_command": "python -m pytest -rA -q",
+ "fail_to_pass": '["t::a"]',
+ "pass_to_pass": ["t::b"],
+ "benchmark": "swe-bench-ext",
+ }
+ ),
+ response=SimpleNamespace(metadata={"model_patch": "diff\n"}, output=[]),
+ )
+ task = build_task(body, "model_patch")
+ assert task.instance_id == "abc"
+ assert task.image == "img:tag"
+ assert task.fail_to_pass == ["t::a"]
+ assert task.pass_to_pass == ["t::b"]
+ assert task.model_patch == "diff\n"
+
+
+def test_build_task_reads_uppercase_fail_pass_to_pass():
+ """Check that build_task falls back to UPPERCASE FAIL_TO_PASS / PASS_TO_PASS keys."""
+ # A row carrying only the UPPERCASE FAIL_TO_PASS / PASS_TO_PASS keys must still
+ # populate the required test lists; otherwise the subset rule (empty subset <= any
+ # set) reports resolved=True for every sample -> reward inflation.
+ body = SimpleNamespace(
+ responses_create_params=SimpleNamespace(
+ metadata={
+ "instance_id": "abc",
+ "FAIL_TO_PASS": '["tests/test_x.py::a"]',
+ "PASS_TO_PASS": ["tests/test_x.py::b"],
+ }
+ ),
+ response=SimpleNamespace(metadata={"model_patch": "diff\n"}, output=[]),
+ )
+ task = build_task(body, "model_patch")
+ assert task.fail_to_pass == ["tests/test_x.py::a"]
+ assert task.pass_to_pass == ["tests/test_x.py::b"]
+
+
+def test_build_task_lowercase_wins_over_uppercase():
+ """Check that lowercase fail_to_pass / pass_to_pass win when both casings are set."""
+ # When both casings are present, lowercase (the canonical key) takes precedence;
+ # the UPPERCASE fallback only fills in when lowercase is absent/empty.
+ body = SimpleNamespace(
+ responses_create_params=SimpleNamespace(
+ metadata={
+ "instance_id": "abc",
+ "fail_to_pass": ["lower::a"],
+ "FAIL_TO_PASS": ["UPPER::a"],
+ "pass_to_pass": ["lower::b"],
+ "PASS_TO_PASS": ["UPPER::b"],
+ }
+ ),
+ response=SimpleNamespace(metadata={"model_patch": "diff\n"}, output=[]),
+ )
+ task = build_task(body, "model_patch")
+ assert task.fail_to_pass == ["lower::a"]
+ assert task.pass_to_pass == ["lower::b"]
+
+
+def test_build_task_unpacks_nested_instance_dict():
+ """Check that build_task surfaces nested instance_dict keys while top-level wins."""
+ # build_task must surface keys nested in a stringified ``instance_dict``
+ # (fail_to_pass_select / base_dockerfile / etc.), with explicit top-level
+ # metadata winning on conflict.
+ body = SimpleNamespace(
+ responses_create_params=SimpleNamespace(
+ metadata={
+ "instance_id": "i",
+ "benchmark": "nv-internal-1",
+ "instance_dict": json.dumps(
+ {
+ "fail_to_pass_select": '["sel::a"]',
+ "base_dockerfile": "ENV FOO=bar",
+ "fail_to_pass": '["nested::a"]',
+ }
+ ),
+ "fail_to_pass": '["toplevel::a"]',
+ }
+ ),
+ response=SimpleNamespace(metadata={"model_patch": "diff\n"}, output=[]),
+ )
+ task = build_task(body, "model_patch")
+ assert task.metadata["fail_to_pass_select"] == '["sel::a"]' # nested key surfaced
+ assert task.metadata["base_dockerfile"] == "ENV FOO=bar"
+ assert task.metadata["fail_to_pass"] == '["toplevel::a"]' # explicit top-level wins
+
+
+def test_item_text_handles_list_content():
+ """Check that _item_text joins the text of a list-valued content field with newlines."""
+ item = SimpleNamespace(content=[SimpleNamespace(text="a"), SimpleNamespace(text="b")])
+ assert _item_text(item) == "a\nb"
+
+
+# ----- reward correctness (FakeSandbox) ---------------------------------------
+
+
+class _FakeProvider:
+ """In-memory sandbox provider that returns canned pytest output for tests."""
+
+ name = "fake-verify"
+
+ def __init__(self, *, test_output="", test_rc=0, **_):
+ """Store the canned test output and return code this provider replays.
+
+ Args:
+ test_output: Stdout returned for any command containing ``pytest``.
+ test_rc: Return code returned for any command containing ``pytest``.
+ """
+ self._test_output = test_output
+ self._test_rc = test_rc
+
+ async def create(self, spec):
+ """Create a fake sandbox handle.
+
+ Args:
+ spec: Sandbox spec whose workdir is echoed back in the handle.
+
+ Returns:
+ SandboxHandle: A handle naming this provider and the spec workdir.
+ """
+ return SandboxHandle(sandbox_id="fake", provider_name=self.name, raw={"workdir": spec.workdir})
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ """Replay the canned output for pytest commands; return success otherwise.
+
+ Args:
+ handle: Sandbox handle (unused).
+ command: Command string; pytest commands replay the canned output.
+ cwd: Working directory (unused).
+ env: Environment variables (unused).
+ timeout_s: Execution timeout (unused).
+ user: User to run as (unused).
+
+ Returns:
+ SandboxExecResult: The canned pytest result, or an empty success result.
+ """
+ if "pytest" in command:
+ return SandboxExecResult(stdout=self._test_output, stderr="", return_code=self._test_rc)
+ return SandboxExecResult(stdout="", stderr="", return_code=0)
+
+ async def upload_file(self, *a, **k):
+ """Accept an upload request and do nothing."""
+ return None
+
+ async def download_file(self, *a, **k):
+ """Accept a download request and do nothing."""
+ return None
+
+ async def status(self, handle):
+ """Report the sandbox as running.
+
+ Args:
+ handle: Sandbox handle (unused).
+
+ Returns:
+ SandboxStatus: Always ``RUNNING``.
+ """
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ """Close the given sandbox handle and do nothing.
+
+ Args:
+ handle: Sandbox handle (unused).
+ """
+ return None
+
+ async def aclose(self):
+ """Close the provider and do nothing."""
+ return None
+
+
+register_provider("fake-verify", _FakeProvider, override=True)
+
+
+def _task(**kw) -> SweTask:
+ """Build a SweTask with sensible defaults, overridable by keyword.
+
+ Args:
+ **kw: Field overrides applied on top of the default task fields.
+
+ Returns:
+ SweTask: The constructed task.
+ """
+ base = dict(
+ instance_id="i",
+ image="img:tag",
+ base_commit="HEAD",
+ test_command="python -m pytest -rA -q",
+ model_patch="diff --git a/x b/x\n",
+ # Trailing-status pytest text (`` PASSED``) is the format the
+ # lighthouse parser recognizes; the ``.py`` path normalizes to this F2P id.
+ test_framework="pytest",
+ fail_to_pass=["tests/test_x.py::a"],
+ benchmark="swe-bench-ext",
+ )
+ base.update(kw)
+ return SweTask(**base)
+
+
+def test_reward_gold_patch_resolves():
+ """Check that a gold patch with passing F2P tests grades to reward 1.0."""
+ report = asyncio.run(verify_task({"fake-verify": {"test_output": "tests/test_x.py::a PASSED\n"}}, _task()))
+ assert reward_from_report(report) == 1.0
+
+
+def test_reward_noop_patch_unresolved():
+ """Check that an empty (no-op) patch grades to reward 0.0."""
+ report = asyncio.run(verify_task({"fake-verify": {}}, _task(model_patch="")))
+ assert reward_from_report(report) == 0.0
+
+
+def test_reward_failing_tests_unresolved():
+ """Check that a patch whose F2P tests fail grades to reward 0.0."""
+ report = asyncio.run(
+ verify_task({"fake-verify": {"test_output": "tests/test_x.py::a FAILED\n", "test_rc": 1}}, _task())
+ )
+ assert reward_from_report(report) == 0.0
+
+
+# ----- REAL docker-backed end-to-end (env-gated; never runs in CI) ------------
+
+_RUN_DOCKER = os.environ.get("SWE_ENV_DOCKER_ITEST") == "1" and shutil.which("docker") is not None
+
+_DOCKERFILE = """FROM python:3.11
+RUN pip install --no-cache-dir pytest
+WORKDIR /testbed
+RUN git config --global user.email a@b.c && git config --global user.name t \\
+ && git init -q \\
+ && printf 'def add(a, b):\\n return a - b\\n' > calc.py \\
+ && printf 'from calc import add\\n\\n\\ndef test_add():\\n assert add(1, 2) == 3\\n' > test_calc.py \\
+ && git add -A && git commit -q -m base
+"""
+
+_GOLD_PATCH = """--- a/calc.py
++++ b/calc.py
+@@ -1,2 +1,2 @@
+ def add(a, b):
+- return a - b
++ return a + b
+"""
+
+_IMAGE_TAG = "swe-env-itest:local"
+
+
+@pytest.mark.skipif(not _RUN_DOCKER, reason="set SWE_ENV_DOCKER_ITEST=1 and install docker to run")
+def test_docker_real_end_to_end():
+ """Build a tiny real git repo image; gold patch resolves, empty patch does not."""
+ build = subprocess.run(
+ ["docker", "build", "-t", _IMAGE_TAG, "-f", "-", "."],
+ input=_DOCKERFILE.encode(),
+ capture_output=True,
+ )
+ assert build.returncode == 0, build.stderr.decode(errors="replace")[-2000:]
+
+ task = SweTask(
+ instance_id="calc-1",
+ image=_IMAGE_TAG,
+ base_commit="HEAD",
+ repo_workdir="/testbed",
+ # Emit JUnit XML to stdout so the lighthouse parser (junit path for
+ # pytest) can grade it host-side.
+ test_command="python -m pytest -q --junit-xml=/dev/stdout",
+ test_framework="pytest",
+ model_patch=_GOLD_PATCH,
+ fail_to_pass=["test_calc.py::test_add"],
+ benchmark="swe-bench-ext",
+ )
+
+ gold_report = asyncio.run(verify_task({"docker": {}}, task))
+ sys.stderr.write(f"\n[itest] gold report: {gold_report}\n")
+ assert gold_report.patch_applied is True
+ assert gold_report.resolved is True
+ assert reward_from_report(gold_report) == 1.0
+
+ import dataclasses
+
+ empty_report = asyncio.run(verify_task({"docker": {}}, dataclasses.replace(task, model_patch="")))
+ assert empty_report.resolved is False
+ assert reward_from_report(empty_report) == 0.0
diff --git a/resources_servers/swe_env/tests/test_verify_http.py b/resources_servers/swe_env/tests/test_verify_http.py
new file mode 100644
index 0000000000..09bb7b5ca9
--- /dev/null
+++ b/resources_servers/swe_env/tests/test_verify_http.py
@@ -0,0 +1,192 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""End-to-end verify() wire contract: an agent POSTs a standard ``BaseVerifyRequest``
+(its response carries the normalized patch) and the verifier returns a non-nullable
+``reward`` plus the eval-side fields, masking via reward=0.0."""
+
+from __future__ import annotations
+
+import asyncio
+
+from nemo_gym.base_resources_server import BaseVerifyRequest
+from nemo_gym.openai_utils import NeMoGymResponse, NeMoGymResponseCreateParamsNonStreaming
+from nemo_gym.sandbox import SandboxExecResult, SandboxHandle, SandboxStatus, register_provider
+from resources_servers.swe_env.app import SweEnvVerifier, SweEnvVerifierConfig
+
+
+class _FakeProvider:
+ """In-memory sandbox provider that replays canned pytest output or fails create."""
+
+ name = "fake-http"
+
+ def __init__(self, *, test_output="", create_error=False, **_):
+ """Store the canned test output and whether sandbox creation should fail.
+
+ Args:
+ test_output: Stdout returned for any command containing ``pytest``.
+ create_error: When True, ``create`` raises ``SandboxCreateError``.
+ """
+ self._test_output = test_output
+ self._create_error = create_error
+
+ async def create(self, spec):
+ """Create a fake sandbox handle, or raise to simulate an infra failure.
+
+ Args:
+ spec: Sandbox spec whose workdir is echoed back in the handle.
+
+ Returns:
+ SandboxHandle: A handle naming this provider and the spec workdir.
+
+ Raises:
+ SandboxCreateError: When this provider was configured with create_error.
+ """
+ if self._create_error:
+ from nemo_gym.sandbox import SandboxCreateError
+
+ raise SandboxCreateError("boom")
+ return SandboxHandle(sandbox_id="h", provider_name=self.name, raw={"workdir": spec.workdir})
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ """Replay the canned output for pytest commands; return success otherwise.
+
+ Args:
+ handle: Sandbox handle (unused).
+ command: Command string; pytest commands replay the canned output.
+ cwd: Working directory (unused).
+ env: Environment variables (unused).
+ timeout_s: Execution timeout (unused).
+ user: User to run as (unused).
+
+ Returns:
+ SandboxExecResult: The canned pytest result, or an empty success result.
+ """
+ if "pytest" in command:
+ return SandboxExecResult(stdout=self._test_output, stderr="", return_code=0)
+ return SandboxExecResult(stdout="", stderr="", return_code=0)
+
+ async def upload_file(self, *a, **k):
+ """Accept an upload request and do nothing."""
+ return None
+
+ async def download_file(self, *a, **k):
+ """Accept a download request and do nothing."""
+ return None
+
+ async def status(self, handle):
+ """Report the sandbox as running.
+
+ Args:
+ handle: Sandbox handle (unused).
+
+ Returns:
+ SandboxStatus: Always ``RUNNING``.
+ """
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ """Close the given sandbox handle and do nothing.
+
+ Args:
+ handle: Sandbox handle (unused).
+ """
+ return None
+
+ async def aclose(self):
+ """Close the provider and do nothing."""
+ return None
+
+
+register_provider("fake-http", _FakeProvider, override=True)
+
+_PATCH = "diff --git a/x b/x\n"
+_METADATA = {
+ "instance_id": "http-e2e",
+ "image": "img:tag",
+ "base_commit": "HEAD",
+ "test_command": "python -m pytest -rA -q",
+ "test_framework": "pytest",
+ "fail_to_pass": '["test_calc.py::test_add"]',
+ "benchmark": "swe-bench-ext",
+}
+
+
+def _request(patch: str) -> BaseVerifyRequest:
+ """Build a verify request whose response metadata carries the given patch.
+
+ Args:
+ patch: The model patch placed in the response's ``model_patch`` metadata.
+
+ Returns:
+ BaseVerifyRequest: A request pairing the standard task metadata with the patch.
+ """
+ params = NeMoGymResponseCreateParamsNonStreaming(
+ input=[{"role": "user", "content": "fix the bug"}], metadata=dict(_METADATA)
+ )
+ response = NeMoGymResponse(
+ id="resp-1",
+ created_at=0,
+ model="m",
+ object="response",
+ output=[],
+ parallel_tool_calls=True,
+ tool_choice="auto",
+ tools=[],
+ metadata={"model_patch": patch},
+ )
+ return BaseVerifyRequest(responses_create_params=params, response=response)
+
+
+def _verifier(provider_cfg) -> SweEnvVerifier:
+ """Build a verifier wired to the given sandbox provider configuration.
+
+ Args:
+ provider_cfg: The sandbox provider config (provider name to kwargs).
+
+ Returns:
+ SweEnvVerifier: A verifier reading the patch from the ``model_patch`` field.
+ """
+ cfg = SweEnvVerifierConfig.model_construct(sandbox_provider=provider_cfg, model_patch_field="model_patch")
+ return SweEnvVerifier.model_construct(config=cfg)
+
+
+def test_verify_returns_reward_for_resolving_patch():
+ """Check that a resolving patch yields reward 1.0 and the eval-side fields."""
+ # Trailing-status pytest text is the format the lighthouse parser recognizes
+ # (the ``.py`` path normalizes to the F2P id in _METADATA).
+ verifier = _verifier({"fake-http": {"test_output": "test_calc.py::test_add PASSED\n"}})
+ out = asyncio.run(verifier.verify(_request(_PATCH)))
+ assert isinstance(out.reward, float)
+ assert out.reward == 1.0
+ assert out.resolved is True
+ assert out.mask_sample is False
+ assert out.instance_id == "http-e2e"
+
+
+def test_verify_masks_infra_error_as_zero_not_none():
+ """Check that an infra failure masks the sample with reward 0.0 rather than None."""
+ verifier = _verifier({"fake-http": {"create_error": True}})
+ out = asyncio.run(verifier.verify(_request(_PATCH)))
+ assert out.reward == 0.0 # never None (non-nullable wire field)
+ assert out.eval_error is True
+ assert out.mask_sample is True
+
+
+def test_verify_empty_patch_unresolved():
+ """Check that an empty patch yields reward 0.0 and patch_exists False."""
+ verifier = _verifier({"fake-http": {}})
+ out = asyncio.run(verifier.verify(_request("")))
+ assert out.reward == 0.0
+ assert out.patch_exists is False
diff --git a/resources_servers/swe_env/verify_task.py b/resources_servers/swe_env/verify_task.py
new file mode 100644
index 0000000000..058640c1e0
--- /dev/null
+++ b/resources_servers/swe_env/verify_task.py
@@ -0,0 +1,155 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Verification orchestrator for the SWE environment verifier.
+
+Imported by the verifier resources server; agents POST a patch to ``/verify``.
+Runs a fresh-only sequence via ``acquire_sandbox`` (always-teardown), bounded by
+a per-call eval timeout. Infra failures are masked as a typed ``error_kind``
+(reward 0.0) rather than crashing the server.
+
+Every eval spec is stamped with a ``ttl_s`` so TTL-honoring backends (such as
+opensandbox) self-expire orphaned sandboxes, and callers bound their ``/verify``
+POST with a client-side timeout so a retried or hung verify cannot pin a rollout
+slot.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import dataclasses
+from collections.abc import Mapping
+from typing import Any
+
+# Importing these packages registers the swe_env providers + harnesses.
+import responses_api_agents.swe_env.harnesses # noqa: F401
+import responses_api_agents.swe_env.providers # noqa: F401
+from nemo_gym.sandbox import SandboxProvider
+from responses_api_agents.swe_env.grading import reward_from_report
+from responses_api_agents.swe_env.harness import SweEvalReport, SweTask
+from responses_api_agents.swe_env.lifecycle import acquire_sandbox
+from responses_api_agents.swe_env.registry import get_harness
+
+
+#: Slack added to the eval timeout when stamping a sandbox TTL (covers spin-up +
+#: teardown so a TTL-honoring backend does not expire a still-running eval).
+_TTL_SLACK_S = 600.0
+
+
+class ProviderCapabilityError(RuntimeError):
+ """Raised when a task's harness does not support the configured provider."""
+
+
+def _provider_name(provider: Mapping[str, Any] | SandboxProvider) -> str:
+ """Return the provider's name.
+
+ Args:
+ provider: Either a single-key provider mapping or a ``SandboxProvider``
+ instance.
+
+ Returns:
+ str: The provider name, or ``"?"`` if it cannot be determined.
+ """
+ if isinstance(provider, Mapping):
+ return next(iter(provider), "?")
+ return getattr(provider, "name", "?")
+
+
+async def verify_task(
+ provider: Mapping[str, Any] | SandboxProvider,
+ task: SweTask,
+ *,
+ run_golden: bool = False,
+ eval_timeout_s: float | None = None,
+) -> SweEvalReport:
+ """Grade a task's patch in a fresh sandbox and return a report.
+
+ Selects the harness for the task's benchmark, optionally substitutes the
+ golden patch, then resets the repo, materializes the patch, runs the eval,
+ and grades the artifacts. An empty patch short-circuits without spinning up
+ a sandbox. Timeouts and infra failures are returned as a report carrying a
+ typed ``error_kind`` rather than raised.
+
+ Args:
+ provider: Single-key provider mapping or ``SandboxProvider`` selecting
+ the sandbox backend.
+ task: The task whose patch is graded.
+ run_golden: When True, grade the task's golden patch instead of the
+ model patch.
+ eval_timeout_s: Optional override for the per-call eval timeout in
+ seconds; falls back to the task metadata or a default.
+
+ Returns:
+ SweEvalReport: The grading outcome, with ``error_kind`` set on timeout
+ or infra failure.
+ """
+ harness = get_harness(task.benchmark)
+
+ if run_golden:
+ task = dataclasses.replace(task, model_patch=task.metadata.get("golden_patch", ""))
+
+ # Empty/falsy-patch fast path: skip eval spin-up entirely.
+ if not (task.model_patch or "").strip():
+ return SweEvalReport(instance_id=task.instance_id, patch_exists=False, resolved=False)
+
+ provider_name = _provider_name(provider)
+ if not harness.supports_provider(provider_name):
+ raise ProviderCapabilityError(
+ f"Harness {harness.name!r} does not support provider {provider_name!r} "
+ f"(grade_strategy={harness.grade_strategy})"
+ )
+
+ spec = harness.build_spec(task)
+ timeout = eval_timeout_s if eval_timeout_s is not None else float(task.metadata.get("eval_timeout_s", 1800))
+ # Stamp a TTL so backends that honor it (opensandbox) self-expire an eval sandbox
+ # orphaned by a hard crash. docker ignores ttl_s; its finally-teardown covers it.
+ if spec.ttl_s is None:
+ spec = dataclasses.replace(spec, ttl_s=timeout + _TTL_SLACK_S)
+
+ try:
+ async with acquire_sandbox(provider, spec, instance_id=task.instance_id) as env:
+
+ async def _sequence() -> SweEvalReport:
+ await harness.reset_repo(env, task)
+ await harness.materialize(env, task)
+ artifacts = await harness.run_eval(env, task)
+ return harness.grade(task, artifacts)
+
+ return await asyncio.wait_for(_sequence(), timeout=timeout)
+ except (asyncio.TimeoutError, TimeoutError):
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ error_kind="eval_timeout",
+ tests_status={"timeout_s": timeout},
+ )
+ except Exception as exc: # infra failure -> mask via flag, never crash the server
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ error_kind="sandbox",
+ tests_status={"exception": repr(exc)},
+ )
+
+
+def report_to_reward(report: SweEvalReport) -> float:
+ """Convert an eval report into a scalar reward.
+
+ Args:
+ report: The grading outcome to score.
+
+ Returns:
+ float: The reward derived from the report.
+ """
+ return reward_from_report(report)
diff --git a/responses_api_agents/swe_env/__init__.py b/responses_api_agents/swe_env/__init__.py
new file mode 100644
index 0000000000..ca1fe89d0f
--- /dev/null
+++ b/responses_api_agents/swe_env/__init__.py
@@ -0,0 +1,50 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Provider-neutral SWE environment library.
+
+Decouples SWE environment infrastructure (sandbox provisioning, exec, and
+verification recipes) from agent harnesses. Built entirely on
+``nemo_gym.sandbox``. Any agent imports this to provision and drive its own
+working container; the separate ``resources_servers/swe_env`` verifier imports
+the harness recipes and grading to score a patch in a fresh sandbox.
+"""
+
+from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+from responses_api_agents.swe_env.grading import compute_resolved, reward_from_report
+from responses_api_agents.swe_env.harness import (
+ EvalArtifacts,
+ SweEvalReport,
+ SweTask,
+ SweTaskHarness,
+)
+from responses_api_agents.swe_env.registry import (
+ get_harness,
+ list_harnesses,
+ register_harness,
+)
+
+
+__all__ = [
+ "AsyncSweEnvironment",
+ "EvalArtifacts",
+ "SweEvalReport",
+ "SweTask",
+ "SweTaskHarness",
+ "compute_resolved",
+ "reward_from_report",
+ "get_harness",
+ "list_harnesses",
+ "register_harness",
+]
diff --git a/responses_api_agents/swe_env/environment.py b/responses_api_agents/swe_env/environment.py
new file mode 100644
index 0000000000..a968f18de5
--- /dev/null
+++ b/responses_api_agents/swe_env/environment.py
@@ -0,0 +1,191 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Async SWE environment adapter over ``nemo_gym.sandbox``.
+
+Provides a thin async wrapper around a sandbox that any agent or the verifier
+can use to run commands and move files in and out of the sandbox.
+"""
+
+from __future__ import annotations
+
+import os
+import tempfile
+from pathlib import Path
+from typing import Any, Mapping
+
+from nemo_gym.sandbox import AsyncSandbox, SandboxProvider, SandboxSpec
+
+
+class AsyncSweEnvironment:
+ """Thin async wrapper around a started ``AsyncSandbox``.
+
+ Agents drive their own loop with ``execute``/``upload``/``download``; the
+ verifier uses the same surface to run eval recipes. The environment never
+ owns trajectory capture or grading logic — only sandbox I/O.
+ """
+
+ def __init__(self, sandbox: AsyncSandbox) -> None:
+ """Wrap an already-started sandbox.
+
+ Args:
+ sandbox (AsyncSandbox): A started sandbox to drive I/O against.
+ """
+ self._sandbox = sandbox
+ self._closed = False
+
+ @classmethod
+ async def start(
+ cls,
+ provider: Mapping[str, Any] | SandboxProvider,
+ spec: SandboxSpec,
+ ) -> "AsyncSweEnvironment":
+ """Create and start a fresh sandbox and return the environment.
+
+ Args:
+ provider (Mapping[str, Any] | SandboxProvider): The sandbox provider
+ config or instance to launch the sandbox with.
+ spec (SandboxSpec): The sandbox spec describing image, workdir, env,
+ and other launch options.
+
+ Returns:
+ AsyncSweEnvironment: An environment wrapping the started sandbox.
+ """
+ sandbox = AsyncSandbox(provider, spec)
+ await sandbox.start()
+ return cls(sandbox)
+
+ @property
+ def sandbox(self) -> AsyncSandbox:
+ """The wrapped sandbox.
+
+ Returns:
+ AsyncSandbox: The underlying sandbox instance.
+ """
+ return self._sandbox
+
+ @property
+ def sandbox_id(self) -> str | None:
+ """The provider-assigned sandbox identifier.
+
+ Returns:
+ str | None: The sandbox id, or ``None`` if the sandbox has no handle.
+ """
+ handle = getattr(self._sandbox, "_handle", None)
+ return handle.sandbox_id if handle is not None else None
+
+ @property
+ def provider_name(self) -> str | None:
+ """The name of the provider backing the sandbox.
+
+ Returns:
+ str | None: The provider name, or ``None`` if the sandbox has no handle.
+ """
+ handle = getattr(self._sandbox, "_handle", None)
+ return handle.provider_name if handle is not None else None
+
+ async def execute(
+ self,
+ command: str,
+ *,
+ cwd: str | None = None,
+ user: str | int | None = "root",
+ timeout_s: int | float | None = None,
+ is_eval: bool = False,
+ ) -> dict[str, Any]:
+ """Run a command in the sandbox and return a normalized result.
+
+ Args:
+ command (str): The shell command to execute.
+ cwd (str | None): Working directory for the command, or ``None`` to
+ use the sandbox default.
+ user (str | int | None): User to run the command as. Defaults to
+ ``"root"``.
+ timeout_s (int | float | None): Optional timeout in seconds.
+ is_eval (bool): Marks the command as part of evaluation.
+
+ Returns:
+ dict[str, Any]: A dict with ``output`` (combined stdout and stderr),
+ ``returncode``, ``stdout``, ``stderr``, and ``error_type``.
+ """
+ result = await self._sandbox.exec(command, cwd=cwd, env=None, timeout_s=timeout_s, user=user)
+ stdout = result.stdout or ""
+ stderr = result.stderr or ""
+ output = "\n".join(part for part in (stdout, stderr) if part)
+ return {
+ "output": output,
+ "returncode": result.return_code,
+ "stdout": stdout,
+ "stderr": stderr,
+ "error_type": result.error_type,
+ }
+
+ async def upload(self, local_path: Path | str, remote_path: str) -> None:
+ """Upload a local file into the sandbox.
+
+ Args:
+ local_path (Path | str): Path to the file on the host.
+ remote_path (str): Destination path inside the sandbox.
+ """
+ await self._sandbox.upload(local_path, remote_path)
+
+ async def download(self, remote_path: str, local_path: Path | str) -> None:
+ """Download a file from the sandbox to the host.
+
+ Args:
+ remote_path (str): Source path inside the sandbox.
+ local_path (Path | str): Destination path on the host.
+ """
+ await self._sandbox.download(remote_path, local_path)
+
+ async def write_text(self, remote_path: str, content: str) -> None:
+ """Write a string to a file inside the sandbox via a temporary upload.
+
+ Args:
+ remote_path (str): Destination path inside the sandbox.
+ content (str): The text content to write.
+ """
+ tmp = tempfile.NamedTemporaryFile("w", delete=False, encoding="utf-8")
+ try:
+ tmp.write(content)
+ tmp.flush()
+ tmp.close()
+ await self._sandbox.upload(tmp.name, remote_path)
+ finally:
+ os.unlink(tmp.name)
+
+ async def cleanup(self) -> None:
+ """Stop the sandbox. Idempotent: subsequent calls are no-ops."""
+ if self._closed:
+ return
+ self._closed = True
+ await self._sandbox.stop()
+
+ async def __aenter__(self) -> "AsyncSweEnvironment":
+ """Enter the async context manager.
+
+ Returns:
+ AsyncSweEnvironment: This environment instance.
+ """
+ return self
+
+ async def __aexit__(self, exc_type: Any, exc_val: Any, exc_tb: Any) -> None:
+ """Exit the async context manager and stop the sandbox.
+
+ Args:
+ exc_type (Any): The exception type, if one was raised.
+ exc_val (Any): The exception instance, if one was raised.
+ exc_tb (Any): The traceback, if an exception was raised.
+ """
+ await self.cleanup()
diff --git a/responses_api_agents/swe_env/grading.py b/responses_api_agents/swe_env/grading.py
new file mode 100644
index 0000000000..5bc53265bf
--- /dev/null
+++ b/responses_api_agents/swe_env/grading.py
@@ -0,0 +1,69 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Pure grading helpers shared by harnesses + the verifier server.
+
+These functions never touch a sandbox; they decide ``resolved`` from parsed
+test status and map a report to a (non-nullable) reward.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Iterable
+
+from responses_api_agents.swe_env.harness import SweEvalReport
+
+
+def compute_resolved(
+ *,
+ fail_to_pass: Iterable[str],
+ pass_to_pass: Iterable[str],
+ passed: Iterable[str],
+) -> bool:
+ """Apply the SWE-bench resolution rule.
+
+ A task is resolved when every FAIL_TO_PASS and PASS_TO_PASS test passes.
+
+ Args:
+ fail_to_pass (Iterable[str]): Tests that must transition from failing to
+ passing.
+ pass_to_pass (Iterable[str]): Tests that must remain passing.
+ passed (Iterable[str]): The tests that actually passed.
+
+ Returns:
+ bool: ``True`` if all required tests passed, ``False`` if there are no
+ required tests or any required test did not pass.
+ """
+ passed_set = set(passed)
+ required = list(fail_to_pass) + list(pass_to_pass)
+ if not required:
+ return False
+ return all(test in passed_set for test in required)
+
+
+def reward_from_report(report: SweEvalReport) -> float:
+ """Map a graded report to a reward.
+
+ An infra or eval failure (``error_kind`` set) yields ``0.0`` and is masked
+ via the flag downstream; the result is always a ``float`` and never ``None``.
+
+ Args:
+ report (SweEvalReport): The graded result to convert.
+
+ Returns:
+ float: ``1.0`` if the task resolved with no error, otherwise ``0.0``.
+ """
+ if report.error_kind is not None:
+ return 0.0
+ return 1.0 if report.resolved else 0.0
diff --git a/responses_api_agents/swe_env/harness.py b/responses_api_agents/swe_env/harness.py
new file mode 100644
index 0000000000..61dddf94db
--- /dev/null
+++ b/responses_api_agents/swe_env/harness.py
@@ -0,0 +1,190 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Task model and harness contract for the SWE environment library.
+
+The harness contract is intentionally split across a trust boundary:
+
+* ``build_spec`` / ``supports_provider`` / ``materialize`` are **provisioning**
+ methods imported and called by *agents* (and the verifier).
+* ``reset_repo`` / ``run_eval`` / ``grade`` are **server-private grading**
+ methods used **only** by the verifier server. A test asserts agent adapters
+ never reference them.
+"""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
+from typing import TYPE_CHECKING, Any
+
+from nemo_gym.sandbox import SandboxSpec
+
+
+if TYPE_CHECKING:
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+
+@dataclass
+class SweTask:
+ """A single SWE task to provision and/or verify.
+
+ Holds the instance metadata needed to launch a sandbox, materialize patches,
+ run the evaluation, and grade the result.
+ """
+
+ instance_id: str
+ image: str | None = None
+ base_commit: str | None = None
+ repo_workdir: str = "/testbed"
+ test_command: str = ""
+ test_framework: str = ""
+ model_patch: str = ""
+ test_patch: str = ""
+ fail_to_pass: list[str] = field(default_factory=list)
+ pass_to_pass: list[str] = field(default_factory=list)
+ benchmark: str = "swe-bench-ext"
+ split: str = "test"
+ metadata: dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass
+class EvalArtifacts:
+ """Raw evaluation output retrieved from the sandbox, before grading."""
+
+ test_output: str = ""
+ return_code: int = 0
+ patch_applied: bool = False
+ raw: dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass
+class SweEvalReport:
+ """Graded result of a single task. ``error_kind`` masks a sample.
+
+ ``error_kind`` is ``None`` for a clean grade. A non-``None`` value (e.g.
+ ``"sandbox"`` / ``"eval_error"``) marks an infra failure: the sample is
+ masked via this flag and ``reward_from_report`` returns ``0.0`` — **never**
+ ``None`` (the wire ``reward`` field is a non-nullable ``float``).
+ """
+
+ instance_id: str
+ resolved: bool = False
+ patch_applied: bool = False
+ patch_exists: bool = False
+ error_kind: str | None = None
+ tests_status: dict[str, Any] = field(default_factory=dict)
+
+
+class SweTaskHarness(ABC):
+ """Per-family provisioning + (server-private) grading recipe."""
+
+ #: registry key, e.g. ``"swe-bench-ext"``.
+ name: str = ""
+ #: ``"flat-host-grade"`` (parse host-side) or ``"nested-harness"`` (in-container grader).
+ grade_strategy: str = "flat-host-grade"
+
+ # --- provisioning (agent-facing + verifier) ------------------------------
+
+ @abstractmethod
+ def build_spec(self, task: SweTask) -> SandboxSpec:
+ """Build the sandbox spec for a task.
+
+ Args:
+ task (SweTask): The task to provision a sandbox for.
+
+ Returns:
+ SandboxSpec: The spec describing image, workdir, env, ttl, and
+ provider options for the task.
+ """
+
+ def supports_provider(self, provider_name: str) -> bool:
+ """Report whether this harness can run on the named provider.
+
+ Nested-Docker families override this to reject exec-only providers.
+
+ Args:
+ provider_name (str): The name of the sandbox provider.
+
+ Returns:
+ bool: ``True`` if the provider is supported.
+ """
+ return True
+
+ async def materialize(self, env: "AsyncSweEnvironment", task: SweTask) -> None:
+ """Upload the model patch and test patch into the started sandbox.
+
+ Args:
+ env (AsyncSweEnvironment): The started environment to write into.
+ task (SweTask): The task whose patches are uploaded.
+ """
+ if task.model_patch:
+ await env.write_text("/root/patch.diff", _ensure_trailing_newline(task.model_patch))
+ if task.test_patch:
+ await env.write_text("/root/test_patch.diff", _ensure_trailing_newline(task.test_patch))
+
+ # --- server-private grading (verifier only) ------------------------------
+
+ async def reset_repo(self, env: "AsyncSweEnvironment", task: SweTask) -> None:
+ """Reset the in-sandbox checkout to ``base_commit`` for hermetic grading.
+
+ Uses only ``git reset --hard``, never ``git clean -fdx``: verification
+ runs in a fresh sandbox (no agent edits to scrub), and a clean would
+ delete the image's prebuilt artifacts (compiled C extensions, installed
+ environment) and break the tests.
+
+ Args:
+ env (AsyncSweEnvironment): The started environment to reset.
+ task (SweTask): The task whose ``base_commit`` and ``repo_workdir``
+ are used.
+ """
+ if task.base_commit:
+ await env.execute(f"git reset --hard {task.base_commit}", cwd=task.repo_workdir)
+
+ @abstractmethod
+ async def run_eval(self, env: "AsyncSweEnvironment", task: SweTask) -> EvalArtifacts:
+ """Apply the patches and run the evaluation, returning raw artifacts.
+
+ Args:
+ env (AsyncSweEnvironment): The started environment to evaluate in.
+ task (SweTask): The task being evaluated.
+
+ Returns:
+ EvalArtifacts: The raw evaluation output retrieved from the sandbox.
+ """
+
+ @abstractmethod
+ def grade(self, task: SweTask, artifacts: EvalArtifacts) -> SweEvalReport:
+ """Parse raw artifacts host-side into a graded report.
+
+ Args:
+ task (SweTask): The task that was evaluated.
+ artifacts (EvalArtifacts): The raw evaluation output to parse.
+
+ Returns:
+ SweEvalReport: The graded result for the task.
+ """
+
+
+def _ensure_trailing_newline(text: str) -> str:
+ """Return the text with a single trailing newline.
+
+ Args:
+ text (str): The input text.
+
+ Returns:
+ str: The text unchanged if it already ends in a newline, otherwise the
+ text with a newline appended.
+ """
+ return text if text.endswith("\n") else text + "\n"
diff --git a/responses_api_agents/swe_env/harnesses/__init__.py b/responses_api_agents/swe_env/harnesses/__init__.py
new file mode 100644
index 0000000000..2f4c5325b8
--- /dev/null
+++ b/responses_api_agents/swe_env/harnesses/__init__.py
@@ -0,0 +1,62 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""SWE dataset-family harnesses. Importing this package registers all families.
+
+Flat host-graded families run on any exec-capable provider (including docker):
+``swe-bench-ext``, ``nv-internal-1``, ``swe-rebench``. Nested-harness families
+run an in-container evaluation and require an apptainer provider, failing fast on
+exec-only providers: ``swe-bench``, ``swe-bench-multilingual``, ``r2e-gym``.
+"""
+
+from responses_api_agents.swe_env.harnesses.nv_internal import NVInternalHarness
+from responses_api_agents.swe_env.harnesses.r2egym import R2EGymHarness
+from responses_api_agents.swe_env.harnesses.swe_bench_ext import SweBenchExtHarness
+from responses_api_agents.swe_env.harnesses.swe_rebench import SweRebenchHarness
+from responses_api_agents.swe_env.harnesses.swebench import SweBenchHarness
+from responses_api_agents.swe_env.registry import list_harnesses, register_harness
+
+
+def register_builtin_harnesses() -> None:
+ """Register every built-in SWE dataset-family harness.
+
+ Constructs each built-in harness and registers it under its name, skipping
+ any name that is already registered so the call is safe to run more than
+ once.
+ """
+ builtins = [
+ SweBenchExtHarness(),
+ NVInternalHarness(),
+ SweRebenchHarness(),
+ SweBenchHarness("swe-bench"),
+ SweBenchHarness("swe-bench-multilingual"),
+ R2EGymHarness(),
+ ]
+ existing = set(list_harnesses())
+ for harness in builtins:
+ if harness.name not in existing:
+ register_harness(harness)
+
+
+register_builtin_harnesses()
+
+
+__all__ = [
+ "NVInternalHarness",
+ "R2EGymHarness",
+ "SweBenchExtHarness",
+ "SweBenchHarness",
+ "SweRebenchHarness",
+ "register_builtin_harnesses",
+]
diff --git a/responses_api_agents/swe_env/harnesses/flat_eval.py b/responses_api_agents/swe_env/harnesses/flat_eval.py
new file mode 100644
index 0000000000..4fd13b1046
--- /dev/null
+++ b/responses_api_agents/swe_env/harnesses/flat_eval.py
@@ -0,0 +1,285 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Flat (host-graded) eval-script mode for SWE dataset families.
+
+Flat mode runs an instance's eval script directly in the sandbox and parses the
+produced log host-side, computing ``resolved`` from ``FAIL_TO_PASS`` /
+``PASS_TO_PASS`` via :func:`compute_resolved`. Because there is no nested
+container, this runs on any exec-capable provider (docker / opensandbox).
+
+The eval script resets the repo, applies the gold/model patch plus the test
+patch, runs the repo's test command, and wraps the test output between two
+sentinel markers::
+
+ >>>>> Start Test Output
+ ... per-test "PASSED " / "FAILED " lines ...
+ >>>>> End Test Output
+
+It also emits patch-apply / reset / timeout status codes
+(``>>>>> Applied Patch`` etc.). The host-side parser in this module recognises
+these markers and per-test status tokens without importing ``swebench``, so
+grading can run in environments where that package (and its Docker
+dependencies) is absent.
+
+Flat mode is selected when the harness is constructed in flat mode
+(``flat_eval=True`` on the constructor) or when a task opts in via
+``SweTask.metadata["flat_eval"]``. Only the harness-level flag lifts the
+apptainer-only ``supports_provider`` restriction; a per-task flag alone affects
+only ``run_eval`` / ``grade`` dispatch on an already-flat-capable harness,
+because the provider is chosen at provisioning time from the harness capability
+before task metadata is consulted.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from responses_api_agents.swe_env.grading import compute_resolved
+from responses_api_agents.swe_env.harness import EvalArtifacts, SweEvalReport, SweTask
+
+
+if TYPE_CHECKING:
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+
+# SWE-bench eval-log sentinels, kept here so we never import swebench at grade
+# time.
+APPLY_PATCH_FAIL = ">>>>> Patch Apply Failed"
+APPLY_PATCH_PASS = ">>>>> Applied Patch"
+RESET_FAILED = ">>>>> Reset Failed"
+TESTS_ERROR = ">>>>> Tests Errored"
+TESTS_TIMEOUT = ">>>>> Tests Timed Out"
+START_TEST_OUTPUT = ">>>>> Start Test Output"
+END_TEST_OUTPUT = ">>>>> End Test Output"
+
+# Codes that mean the harness/patch/test setup failed before tests could be
+# trusted; their presence forces an empty status map + patch_applied=False.
+_BAD_CODES = (APPLY_PATCH_FAIL, RESET_FAILED, TESTS_ERROR, TESTS_TIMEOUT)
+
+# Per-test status tokens a pytest-style test runner emits at the start of a line
+# ("PASSED tests/test_x.py::test_a"). XFAIL counts as a pass.
+_PASS_TOKENS = ("PASSED", "XFAIL")
+_FAIL_TOKENS = ("FAILED", "ERROR")
+_STATUS_TOKENS = _PASS_TOKENS + _FAIL_TOKENS + ("SKIPPED",)
+
+# Where the flat path writes the eval script and its captured log inside the
+# sandbox.
+EVAL_SCRIPT_PATH = "/root/eval.sh"
+EVAL_LOG_PATH = "/root/eval_output.log"
+
+
+def parse_eval_log(log: str) -> tuple[dict[str, str], bool]:
+ """Parse a SWE-bench eval-script log host-side.
+
+ For the common pytest-style runner:
+
+ 1. If any "bad code" (patch-apply / reset / tests-error / timeout) is
+ present, the run is untrustworthy -> return ``({}, False)``.
+ 2. If the ``Start``/``End`` test-output markers are missing, the test patch
+ never applied -> return ``({}, False)``.
+ 3. Otherwise extract the slice between the markers and parse per-test
+ ``" "`` lines into a ``{node_id: STATUS}`` map. As a
+ fallback (output sometimes escapes the markers, e.g. to stderr) the whole
+ log is scanned when the slice yields nothing.
+
+ Args:
+ log: The combined stdout/stderr captured from running the eval script.
+
+ Returns:
+ A tuple ``(status_map, patch_applied)``. ``status_map`` maps each test
+ node id to its status token. ``patch_applied`` is ``True`` only when the
+ markers were found and no bad code fired.
+ """
+ if any(code in log for code in _BAD_CODES):
+ return {}, False
+ if START_TEST_OUTPUT not in log or END_TEST_OUTPUT not in log:
+ return {}, False
+
+ between = log.split(START_TEST_OUTPUT, 1)[1].split(END_TEST_OUTPUT, 1)[0]
+ status_map = _parse_pytest_status_lines(between)
+ if not status_map:
+ # Fallback: some runners emit per-test lines outside the markers.
+ status_map = _parse_pytest_status_lines(log)
+ return status_map, True
+
+
+def _parse_pytest_status_lines(text: str) -> dict[str, str]:
+ """Parse ``" "`` pytest-style lines into a status map.
+
+ A status line starts with one of the recognised status tokens, and the node
+ id is the second whitespace field. FAILED lines may read
+ ``"FAILED - "``; the trailing reason is stripped by rewriting
+ ``" - "`` to ``" "``.
+
+ Args:
+ text: Text containing zero or more per-test status lines.
+
+ Returns:
+ A mapping from each test node id to its status token. When a node id
+ appears more than once, the last occurrence wins.
+ """
+ status_map: dict[str, str] = {}
+ for raw_line in text.split("\n"):
+ line = raw_line.strip()
+ token = next((t for t in _STATUS_TOKENS if line.startswith(t)), None)
+ if token is None:
+ continue
+ if token == "FAILED":
+ line = line.replace(" - ", " ")
+ fields = line.split()
+ if len(fields) <= 1:
+ continue
+ node_id = fields[1]
+ # Last status wins for a duplicated node id: a later line overwrites an
+ # earlier one, so a runner that re-reports a node (e.g. a rerun plugin)
+ # ends up with its final status.
+ status_map[node_id] = fields[0]
+ return status_map
+
+
+def passed_tests(status_map: dict[str, str]) -> list[str]:
+ """Return node ids whose status counts as a pass (PASSED or XFAIL).
+
+ Args:
+ status_map: A mapping from test node id to its status token.
+
+ Returns:
+ The list of node ids whose status is a passing token.
+ """
+ return [node for node, status in status_map.items() if status in _PASS_TOKENS]
+
+
+async def flat_run_eval(env: "AsyncSweEnvironment", task: SweTask) -> EvalArtifacts:
+ """Run the instance's eval script in the sandbox and capture its log.
+
+ The eval script must be supplied on the task via
+ ``task.metadata["eval_script"]``. It is written into the sandbox and run,
+ teeing its combined output to :data:`EVAL_LOG_PATH`; the captured
+ stdout/stderr already contain the ``>>>>>`` markers, so ``test_output`` is
+ graded directly. The log file is read back as a fallback when the streamed
+ output is empty.
+
+ Args:
+ env: The SWE environment used to write files and execute commands in the
+ sandbox.
+ task: The task whose ``metadata["eval_script"]`` is run.
+
+ Returns:
+ An :class:`EvalArtifacts` holding the captured test output, the script's
+ return code, whether a model patch existed, and raw metadata. When no
+ eval script is present the artifacts carry an ``eval_error``.
+ """
+ eval_script = task.metadata.get("eval_script", "")
+ if not eval_script:
+ # No script to run -> mask as an eval error rather than scoring 0.
+ return EvalArtifacts(
+ test_output="",
+ return_code=1,
+ patch_applied=False,
+ raw={"error_type": "eval_error", "flat": True},
+ )
+
+ await env.write_text(EVAL_SCRIPT_PATH, eval_script if eval_script.endswith("\n") else eval_script + "\n")
+ # The script is self-contained (it resets + applies patches + runs tests);
+ # `|| true` keeps the captured log even on a non-zero test exit so grade()
+ # can parse per-test status. Combined output is also tee'd to a log file.
+ result = await env.execute(
+ f"bash {EVAL_SCRIPT_PATH} 2>&1 | tee {EVAL_LOG_PATH}; exit ${{PIPESTATUS[0]}}",
+ cwd=task.repo_workdir,
+ is_eval=True,
+ timeout_s=task.metadata.get("tests_timeout"),
+ )
+ log_text = result["output"]
+ if not log_text.strip() and result.get("error_type") not in {"sandbox", "timeout"}:
+ # Streamed output was empty; fall back to the tee'd log file.
+ cat = await env.execute(f"cat {EVAL_LOG_PATH}", cwd=task.repo_workdir)
+ if cat["returncode"] == 0:
+ log_text = cat["output"]
+
+ return EvalArtifacts(
+ test_output=log_text,
+ return_code=result["returncode"],
+ patch_applied=bool(task.model_patch),
+ raw={"error_type": result.get("error_type"), "flat": True},
+ )
+
+
+def flat_grade(task: SweTask, artifacts: EvalArtifacts) -> SweEvalReport:
+ """Grade a flat eval-script log host-side.
+
+ Infra failures (sandbox/timeout) are masked via ``error_kind``. A log with a
+ bad code or missing markers grades as unresolved with ``patch_applied`` set
+ from the parse, since a failed setup is a legitimate unresolved rather than
+ an infra mask.
+
+ Args:
+ task: The task being graded, supplying the instance id, expected
+ ``fail_to_pass`` / ``pass_to_pass`` tests, and model patch.
+ artifacts: The eval artifacts produced by :func:`flat_run_eval`.
+
+ Returns:
+ A :class:`SweEvalReport` describing whether the task was resolved,
+ whether the patch applied and existed, any masking ``error_kind``, and
+ the per-test status breakdown.
+ """
+ if artifacts.raw.get("error_type") in {"sandbox", "timeout"}:
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ patch_applied=artifacts.patch_applied,
+ error_kind=artifacts.raw["error_type"],
+ )
+ # A missing eval script is an eval error (masked), not a 0 score.
+ if artifacts.raw.get("error_type") == "eval_error":
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ patch_applied=artifacts.patch_applied,
+ error_kind="eval_error",
+ )
+
+ status_map, log_patch_applied = parse_eval_log(artifacts.test_output)
+ passed = passed_tests(status_map)
+ resolved = log_patch_applied and compute_resolved(
+ fail_to_pass=task.fail_to_pass,
+ pass_to_pass=task.pass_to_pass,
+ passed=passed,
+ )
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ resolved=resolved,
+ patch_applied=log_patch_applied,
+ patch_exists=bool(task.model_patch),
+ tests_status={"passed": passed, "all": status_map},
+ )
+
+
+def flat_eval_enabled(harness_flag: bool, task: SweTask) -> bool:
+ """Return whether flat mode should be used for this task.
+
+ Flat mode is selected when the harness was constructed in flat mode
+ (``harness_flag``) or the task opts in via ``metadata["flat_eval"]``. The
+ harness flag lifts the ``supports_provider`` apptainer-only gate; the
+ per-task key only affects ``run_eval`` / ``grade`` dispatch on an
+ already-flat-capable harness.
+
+ Args:
+ harness_flag: Whether the harness instance was constructed in flat mode.
+ task: The task whose ``metadata["flat_eval"]`` is consulted.
+
+ Returns:
+ ``True`` when flat mode applies to this task, otherwise ``False``.
+ """
+ return bool(harness_flag) or bool(task.metadata.get("flat_eval", False))
diff --git a/responses_api_agents/swe_env/harnesses/nv_internal.py b/responses_api_agents/swe_env/harnesses/nv_internal.py
new file mode 100644
index 0000000000..4664093dca
--- /dev/null
+++ b/responses_api_agents/swe_env/harnesses/nv_internal.py
@@ -0,0 +1,426 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""nv-internal-1 harness: flat, host-graded NVIDIA-internal family.
+
+This family does not run any in-container grading harness: it ships a per-instance
+``run_script.sh`` + ``parsing_script.py`` that emit a structured ``output.json``
+test report. The recipe is a 3-hop sequence:
+
+ 1. ``bash run_script.sh > stdout.log 2> stderr.log`` (keep streams separate)
+ 2. ``python parsing_script.py stdout.log stderr.log output.json`` (parse to JSON report)
+ 3. read ``output.json`` back host-side
+
+Grading is then a pure host-side parse of that report's ``{tests: [{name, status}]}``
+shape. Because the family is flat and host-graded, it runs on any exec-capable
+provider (e.g. docker). The run script, parsing script, and model patch are
+uploaded by ``materialize``.
+"""
+
+from __future__ import annotations
+
+import ast
+import json
+import re
+from typing import TYPE_CHECKING, Any
+
+from nemo_gym.sandbox import SandboxResources, SandboxSpec
+from responses_api_agents.swe_env.grading import compute_resolved
+from responses_api_agents.swe_env.harness import (
+ EvalArtifacts,
+ SweEvalReport,
+ SweTask,
+ SweTaskHarness,
+ _ensure_trailing_newline,
+)
+
+
+if TYPE_CHECKING:
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+
+#: nv-internal default working directory.
+NV_DEFAULT_WORKDIR = "/app"
+#: The generic ``build_task`` default workdir; means "the row didn't set one".
+_GENERIC_DEFAULT_WORKDIR = "/testbed"
+
+
+def _nv_workdir(task: SweTask) -> str:
+ """Resolve the working directory for nv-internal hops.
+
+ The generic ``build_task`` defaults ``repo_workdir`` to ``/testbed``, which is
+ not the nv-internal convention. A row that explicitly sets a non-default
+ ``repo_workdir`` is honored; otherwise the nv-internal default ``/app`` is used.
+
+ Args:
+ task: The task whose ``repo_workdir`` is consulted.
+
+ Returns:
+ The working directory path (str) to run every nv-internal hop in.
+ """
+ workdir = task.repo_workdir
+ if not workdir or workdir == _GENERIC_DEFAULT_WORKDIR:
+ return NV_DEFAULT_WORKDIR
+ return workdir
+
+
+def parse_passed_tests(report: dict[str, Any]) -> list[str]:
+ """Extract PASSED test names from a parsing_script ``output.json`` report.
+
+ The report shape is ``{"tests": [{"name": ..., "status": "PASSED"|...}, ...]}``.
+
+ Args:
+ report: The parsed ``output.json`` report mapping.
+
+ Returns:
+ The list of test names (list[str]) whose status is ``"PASSED"``.
+ """
+ return [
+ test["name"]
+ for test in report.get("tests", [])
+ if isinstance(test, dict) and test.get("status") == "PASSED" and "name" in test
+ ]
+
+
+class NVInternalHarness(SweTaskHarness):
+ """Flat, host-graded harness for the NVIDIA-internal task family.
+
+ Tasks ship their own ``run_script.sh`` and ``parsing_script.py`` that produce
+ a structured ``output.json`` report, which is graded entirely host-side. The
+ harness runs on any exec-capable provider.
+ """
+
+ name = "nv-internal-1"
+ grade_strategy = "flat-host-grade"
+
+ def build_spec(self, task: SweTask) -> SandboxSpec:
+ """Build the sandbox spec for an nv-internal task.
+
+ Environment variables parsed from the task's dockerfiles are injected into
+ ``spec.env`` so the provider applies them to every exec hop. This is a
+ no-op when the dataset does not carry the dockerfiles.
+
+ Args:
+ task: The task to build a sandbox spec for.
+
+ Returns:
+ A :class:`SandboxSpec` describing the image, workdir, timeouts,
+ environment, metadata, resources, and provider options.
+ """
+ env = {"GIT_CONFIG_GLOBAL": "/dev/null", "GIT_PAGER": "cat"}
+ env.update(_parse_dockerfile_env(task))
+ return SandboxSpec(
+ image=task.image,
+ workdir=_nv_workdir(task),
+ ttl_s=task.metadata.get("ttl_s", 1800),
+ ready_timeout_s=task.metadata.get("ready_timeout_s", 600),
+ env=env,
+ metadata={
+ "instance_id": task.instance_id[:63],
+ "benchmark": task.benchmark,
+ "harness": self.name,
+ },
+ resources=SandboxResources.from_mapping(task.metadata.get("resources", {})),
+ provider_options=task.metadata.get("provider_options", {}),
+ )
+
+ def supports_provider(self, provider_name: str) -> bool:
+ """Report whether this harness supports the named provider.
+
+ The family is flat and host-graded, so every exec-capable provider is
+ supported.
+
+ Args:
+ provider_name: The provider name being checked.
+
+ Returns:
+ ``True`` for every provider.
+ """
+ return True # flat, host-graded: works on any exec-capable provider
+
+ async def materialize(self, env: "AsyncSweEnvironment", task: SweTask) -> None:
+ """Upload run_script.sh, parsing_script.py, and the model patch.
+
+ The scripts live in ``task.metadata``. The dataset stores them under
+ dotted keys (``"run_script.sh"`` / ``"parsing_script.py"``), which are read
+ first, falling back to the extensionless keys only if the dotted ones are
+ absent.
+
+ Args:
+ env: The environment used to write files into the sandbox.
+ task: The task carrying the patch and scripts to upload.
+ """
+ if task.model_patch:
+ await env.write_text("/root/patch.diff", _ensure_trailing_newline(task.model_patch))
+ run_script = task.metadata.get("run_script.sh") or task.metadata.get("run_script", "")
+ parsing_script = task.metadata.get("parsing_script.py") or task.metadata.get("parsing_script", "")
+ if run_script:
+ await env.write_text("/root/run_script.sh", _ensure_trailing_newline(run_script))
+ if parsing_script:
+ await env.write_text("/root/parsing_script.py", _ensure_trailing_newline(parsing_script))
+
+ async def reset_repo(self, env: "AsyncSweEnvironment", task: SweTask) -> None:
+ """Reset the checkout to ``base_commit``.
+
+ Runs ``git reset --hard`` followed by ``git checkout`` of the base commit
+ (not ``git clean``) in the nv-internal working directory.
+
+ Args:
+ env: The environment used to execute commands in the sandbox.
+ task: The task carrying the ``base_commit`` to reset to.
+ """
+ if task.base_commit:
+ await env.execute(
+ f"git reset --hard {task.base_commit} && git checkout {task.base_commit}",
+ cwd=_nv_workdir(task),
+ )
+
+ async def run_eval(self, env: "AsyncSweEnvironment", task: SweTask) -> EvalArtifacts:
+ """Run the 3-hop evaluation recipe and collect its artifacts.
+
+ Applies the model patch, runs the optional per-instance repo setup hook,
+ then executes the run/parse/read sequence. Sandbox or timeout failures in
+ any hop short-circuit and are surfaced via ``raw["error_type"]``.
+
+ Args:
+ env: The environment used to execute commands in the sandbox.
+ task: The task being evaluated.
+
+ Returns:
+ An :class:`EvalArtifacts` holding the report output, return code,
+ whether the patch applied cleanly, and any infra error type.
+ """
+ workdir = _nv_workdir(task)
+ # Apply the model patch with rejection to tolerate conflicts:
+ # `--reject` writes .rej files instead of failing; `|| true` keeps going.
+ patch_applied = True
+ if task.model_patch:
+ applied = await env.execute(
+ "git apply --ignore-space-change --ignore-whitespace --reject -v /root/patch.diff",
+ cwd=workdir,
+ )
+ patch_applied = applied["returncode"] == 0
+
+ # Optional per-instance repo setup hook.
+ repo_cmd = task.metadata.get("before_repo_set_cmd", "").strip()
+ if repo_cmd:
+ repo_cmd = repo_cmd.split("\n")[-1]
+ setup = await env.execute(repo_cmd, cwd=workdir, is_eval=True)
+ if setup.get("error_type") in {"sandbox", "timeout"}:
+ return EvalArtifacts(
+ test_output=setup["output"],
+ return_code=setup["returncode"],
+ patch_applied=patch_applied,
+ raw={"error_type": setup.get("error_type")},
+ )
+
+ # Hop 1: run the per-instance script, keeping stdout/stderr separate.
+ # The selected test files are passed positionally.
+ test_files = _format_test_files(task.metadata.get("selected_test_files_to_run", []))
+ run = await env.execute(
+ f"bash /root/run_script.sh {test_files} > /root/stdout.log 2> /root/stderr.log || true",
+ cwd=workdir,
+ is_eval=True,
+ )
+ if run.get("error_type") in {"sandbox", "timeout"}:
+ return EvalArtifacts(
+ test_output=run["output"],
+ return_code=run["returncode"],
+ patch_applied=patch_applied,
+ raw={"error_type": run.get("error_type")},
+ )
+
+ # Hop 2: parse the logs into a JSON report.
+ parse = await env.execute(
+ "python /root/parsing_script.py /root/stdout.log /root/stderr.log /root/output.json",
+ cwd=workdir,
+ is_eval=True,
+ )
+ if parse.get("error_type") in {"sandbox", "timeout"}:
+ return EvalArtifacts(
+ test_output=parse["output"],
+ return_code=parse["returncode"],
+ patch_applied=patch_applied,
+ raw={"error_type": parse.get("error_type")},
+ )
+
+ # Hop 3: read the report back host-side.
+ report = await env.execute("cat /root/output.json", cwd=workdir, is_eval=True)
+ return EvalArtifacts(
+ test_output=report["output"],
+ return_code=report["returncode"],
+ patch_applied=patch_applied,
+ raw={"error_type": report.get("error_type")},
+ )
+
+ def grade(self, task: SweTask, artifacts: EvalArtifacts) -> SweEvalReport:
+ """Grade the evaluation artifacts into a report.
+
+ Parses the host-side ``output.json`` report, extracts PASSED tests, and
+ derives resolution from the required FAIL_TO_PASS / PASS_TO_PASS sets. An
+ infra failure (sandbox or timeout) is masked via ``error_kind`` rather than
+ scored as unresolved.
+
+ Args:
+ task: The task being graded.
+ artifacts: The artifacts produced by ``run_eval``.
+
+ Returns:
+ A :class:`SweEvalReport` with resolution status, patch flags, and the
+ parsed test report.
+ """
+ # Infra failure → mask via error_kind (never scored as "unresolved").
+ if artifacts.raw.get("error_type") in {"sandbox", "timeout"}:
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ patch_applied=artifacts.patch_applied,
+ error_kind=artifacts.raw["error_type"],
+ )
+ try:
+ report = json.loads(artifacts.test_output) if artifacts.test_output.strip() else {}
+ except (ValueError, TypeError):
+ report = {}
+ passed = parse_passed_tests(report)
+ f2p, p2p = _resolve_required_tests(task)
+ # Resolution is derived from tests alone and never gated on patch-apply rc.
+ # An empty report or no required tests → unresolved (compute_resolved
+ # returns False).
+ resolved = compute_resolved(
+ fail_to_pass=f2p,
+ pass_to_pass=p2p,
+ passed=passed,
+ )
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ resolved=resolved,
+ patch_applied=artifacts.patch_applied,
+ patch_exists=bool(task.model_patch),
+ tests_status={"passed": passed, "report": report},
+ )
+
+
+def _format_test_files(test_files: Any) -> str:
+ """Build the comma-joined test-files argument.
+
+ Accepts a list, or a string that is either a comma-joined value or a
+ ``repr``-style list. A stringified list may use single quotes
+ (``['a', 'b']``) which ``json.loads`` rejects, so ``ast.literal_eval`` is used
+ (handling single-quoted and native lists) with a safe fallback to the raw
+ string.
+
+ Args:
+ test_files: A list/tuple of names, or a string holding a comma-joined
+ value or a stringified list.
+
+ Returns:
+ The comma-joined test-files argument (str); empty for unsupported inputs.
+ """
+ if isinstance(test_files, (list, tuple)):
+ return ",".join(str(item) for item in test_files)
+ if isinstance(test_files, str):
+ stripped = test_files.strip()
+ if stripped.startswith("[") and stripped.endswith("]"):
+ try:
+ parsed = ast.literal_eval(stripped)
+ if isinstance(parsed, (list, tuple)):
+ return ",".join(str(item) for item in parsed)
+ except (ValueError, SyntaxError):
+ pass
+ return stripped
+ return ""
+
+
+def _resolve_required_tests(task: SweTask) -> tuple[list[str], list[str]]:
+ """Resolve the FAIL_TO_PASS / PASS_TO_PASS required-test sets.
+
+ The ``fail_to_pass_select`` / ``pass_to_pass_select`` keys on ``task.metadata``
+ take precedence when present; otherwise the plain ``task.fail_to_pass`` /
+ ``task.pass_to_pass`` are used. Values may be lists or stringified lists.
+
+ Args:
+ task: The task whose required-test sets are resolved.
+
+ Returns:
+ A ``(fail_to_pass, pass_to_pass)`` tuple of test-name lists.
+ """
+ f2p = task.metadata.get("fail_to_pass_select")
+ f2p = _coerce_test_list(f2p) if f2p is not None else list(task.fail_to_pass)
+ p2p = task.metadata.get("pass_to_pass_select")
+ p2p = _coerce_test_list(p2p) if p2p is not None else list(task.pass_to_pass)
+ return f2p, p2p
+
+
+def _coerce_test_list(value: Any) -> list[str]:
+ """Coerce a test-list value (list or stringified list) into a list of names.
+
+ Args:
+ value: A list/tuple of names, or a string holding a stringified list.
+
+ Returns:
+ The list of test names (list[str]); empty for unsupported inputs.
+ """
+ if isinstance(value, (list, tuple)):
+ return [str(item) for item in value]
+ if isinstance(value, str):
+ stripped = value.strip()
+ if stripped.startswith("[") and stripped.endswith("]"):
+ try:
+ parsed = ast.literal_eval(stripped)
+ if isinstance(parsed, (list, tuple)):
+ return [str(item) for item in parsed]
+ except (ValueError, SyntaxError):
+ pass
+ return []
+
+
+def _parse_dockerfile_env(task: SweTask) -> dict[str, str]:
+ """Parse ``ENV`` lines from the task's dockerfiles into a name->value mapping.
+
+ Scans ``base_dockerfile + instance_dockerfile`` for ``ENV`` directives and
+ converts them to environment variables. Handles both Docker forms:
+
+ ENV KEY=VALUE (equals)
+ ENV KEY VALUE (space-separated)
+
+ Returns ``{}`` when the dockerfiles are absent from metadata.
+
+ Args:
+ task: The task whose dockerfile metadata is scanned.
+
+ Returns:
+ A mapping (dict[str, str]) of environment variable names to values.
+ """
+ base_dockerfile = str(task.metadata.get("base_dockerfile", "") or "")
+ instance_dockerfile = str(task.metadata.get("instance_dockerfile", "") or "")
+ env: dict[str, str] = {}
+ for raw_line in (base_dockerfile + "\n" + instance_dockerfile).split("\n"):
+ line = raw_line.strip()
+ if not line.startswith("ENV "):
+ continue
+ body = line[len("ENV ") :].strip()
+ if "=" in body:
+ # Format: ENV KEY=VALUE -> normalize spaces around the first `=`.
+ key, _, value = body.partition("=")
+ key = re.sub(r"\s+", "", key)
+ value = value.strip()
+ else:
+ # Format: ENV KEY VALUE -> split into key + remainder value.
+ parts = body.split(None, 1)
+ if len(parts) < 2:
+ continue
+ key, value = parts[0], parts[1]
+ if key:
+ env[key] = value
+ return env
diff --git a/responses_api_agents/swe_env/harnesses/r2egym.py b/responses_api_agents/swe_env/harnesses/r2egym.py
new file mode 100644
index 0000000000..26d8959388
--- /dev/null
+++ b/responses_api_agents/swe_env/harnesses/r2egym.py
@@ -0,0 +1,351 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""r2e-gym harness: nested, in-container-graded family.
+
+Unlike the flat ``swe-bench-ext`` family, r2e-gym does NOT grade host-side: the
+per-instance ``report.json`` is produced by the vendored r2e-gym evaluation
+harness (``run_local_evaluation.py``) running inside the container. ``grade()``
+therefore only parses that report's already-computed ``resolved`` verdict rather
+than reconstructing it from per-test status.
+
+Two r2e-gym-specific wrinkles:
+
+* **Test hiding during the agent phase.** ``/r2e_tests`` holds the held-out
+ evaluation tests, and ``run_tests.sh`` launches them, so both are removed from
+ the agent's checkout (root, ``/root``, ``/testbed``). During grading (the
+ verifier) these are present, because the nested harness re-materializes them —
+ ``hide_eval_tests_commands`` is exposed for the agent adapter to run after
+ ``materialize`` and is intentionally NOT invoked by ``run_eval``.
+* **r2egym_setup mount.** The prebuilt R2E-Gym venv has hardcoded absolute paths
+ in its uv wrappers, so the setup dir is bind-mounted at both ``/r2egym_setup``
+ and its original absolute path. These mounts are surfaced via
+ ``provider_options["mounts"]`` for the apptainer provider.
+
+This family requires apptainer and a real ``.sif`` container; it cannot run on
+exec-only / docker providers. ``supports_provider`` fails fast on any
+non-apptainer provider.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import TYPE_CHECKING
+
+from nemo_gym.sandbox import SandboxResources, SandboxSpec
+from responses_api_agents.swe_env.harness import (
+ EvalArtifacts,
+ SweEvalReport,
+ SweTask,
+ SweTaskHarness,
+ _ensure_trailing_newline,
+)
+from responses_api_agents.swe_env.harnesses import flat_eval
+
+
+if TYPE_CHECKING:
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+
+# Location the nested r2e-gym harness writes its per-instance report to inside
+# the container. ``run_eval`` redirects ``run_local_evaluation.py`` here and
+# then reads it back host-side for parsing.
+_REPORT_PATH = "/root/r2egym_report.json"
+
+# In-container predictions JSONL the nested harness reads via
+# ``--predictions_path``. Holds the SWE-bench predictions shape
+# ({instance_id, model_patch, ...}).
+_PREDICTIONS_PATH = "/root/predictions.jsonl"
+
+
+class R2EGymHarness(SweTaskHarness):
+ """Harness for the r2e-gym family of SWE tasks.
+
+ Grades by parsing the report produced by the nested r2e-gym evaluation
+ harness running inside the container, or, in opt-in flat mode, by running
+ the instance's eval script in-sandbox and parsing the log host-side.
+ """
+
+ name = "r2e-gym"
+ grade_strategy = "nested-harness"
+
+ def __init__(self, *, flat_eval: bool = False) -> None:
+ """Initialize the harness.
+
+ Args:
+ flat_eval: When True, opt into flat (host-graded) mode: run the
+ instance's eval script directly in the sandbox and parse the log
+ host-side, lifting the apptainer-only gate so it can run on any
+ exec-capable provider. When False (default), keep the nested
+ in-container grading behavior.
+ """
+ self.flat_eval = flat_eval
+ if flat_eval:
+ self.grade_strategy = "flat-host-grade"
+
+ def build_spec(self, task: SweTask) -> SandboxSpec:
+ """Build the sandbox spec for an r2e-gym task.
+
+ Bind-mounts the r2e-gym setup dir at both ``/r2egym_setup`` and its
+ original absolute path so the prebuilt uv venv (which has hardcoded
+ absolute toolchain paths) resolves correctly, surfacing the mounts via
+ ``provider_options["mounts"]`` for the apptainer provider.
+
+ Args:
+ task: The SWE task whose metadata, image, and workdir describe the
+ sandbox to construct.
+
+ Returns:
+ SandboxSpec: The fully populated sandbox spec, including image,
+ workdir, TTL, environment, metadata, resources, and provider
+ options with the r2e-gym setup mounts.
+ """
+ setup_dir = task.metadata.get("r2egym_setup_dir", "/r2egym_setup")
+ # The prebuilt uv venv has hardcoded absolute paths, so the setup dir is
+ # bind-mounted at both ``/r2egym_setup`` and its original absolute path.
+ # These are surfaced via ``provider_options['mounts']`` so the prebuilt
+ # ``{setup}/R2E-Gym/venv`` is bound in and resolves its hardcoded
+ # toolchain paths.
+ mounts = [
+ {"src": setup_dir, "dst": "/r2egym_setup"},
+ {"src": setup_dir, "dst": setup_dir},
+ ]
+ provider_options = dict(task.metadata.get("provider_options", {}))
+ provider_options.setdefault("mounts", mounts)
+ return SandboxSpec(
+ image=task.image,
+ workdir=task.repo_workdir,
+ ttl_s=task.metadata.get("ttl_s", 1800),
+ ready_timeout_s=task.metadata.get("ready_timeout_s", 600),
+ env={"GIT_CONFIG_GLOBAL": "/dev/null", "GIT_PAGER": "cat"},
+ metadata={
+ "instance_id": task.instance_id[:63],
+ "benchmark": task.benchmark,
+ "harness": self.name,
+ },
+ resources=SandboxResources.from_mapping(task.metadata.get("resources", {})),
+ provider_options=provider_options,
+ )
+
+ def supports_provider(self, provider_name: str) -> bool:
+ """Report whether this harness can run on the named sandbox provider.
+
+ Args:
+ provider_name: Name of the sandbox provider (e.g. ``"apptainer"``,
+ ``"docker"``, ``"local"``).
+
+ Returns:
+ bool: True for any provider in flat mode (host-graded, no nested
+ container); otherwise True only for ``"apptainer"``, since the
+ nested vendored harness requires apptainer with a real ``.sif``.
+ """
+ # Flat mode is host-graded (no nested container), so it runs on any
+ # exec-capable provider.
+ if self.flat_eval:
+ return True
+ # Nested family: the vendored harness only runs under apptainer with a
+ # real .sif. Fail fast on exec-only providers (docker/local).
+ return provider_name == "apptainer"
+
+ async def materialize(self, env: "AsyncSweEnvironment", task: SweTask) -> None:
+ """Write the model patch into the sandbox for the nested grader.
+
+ The nested harness reads the model patch from a predictions JSONL keyed
+ by instance_id (``run_local_evaluation.py --predictions_path``), in the
+ SWE-bench predictions shape. The patch's trailing newline is normalized
+ before embedding: a non-empty patch missing its trailing newline gets one
+ appended; an empty/absent patch stays empty. ``git apply`` is
+ newline-sensitive, so an unnormalized patch can fail to apply and silently
+ flip ``resolved`` to False.
+
+ Args:
+ env: The active SWE environment used to write files into the sandbox.
+ task: The SWE task supplying the model patch, instance id, and
+ metadata.
+ """
+ patch = task.model_patch or ""
+ prediction = {
+ "instance_id": task.instance_id,
+ "model_name_or_path": task.metadata.get("model_name_or_path", "nemo-gym"),
+ "model_patch": _ensure_trailing_newline(patch) if patch else "",
+ }
+ await env.write_text(_PREDICTIONS_PATH, json.dumps(prediction) + "\n")
+
+ async def reset_repo(self, env: "AsyncSweEnvironment", task: SweTask) -> None:
+ """Reset the repository checkout (no-op for r2e-gym).
+
+ No host-orchestrated reset is performed for this family: the nested
+ ``run_local_evaluation`` resets the checkout inside its own container.
+ Running a host-side ``git reset --hard `` here would mutate
+ state the nested grader owns.
+
+ Args:
+ env: The active SWE environment (unused).
+ task: The SWE task (unused).
+ """
+ return None
+
+ def hide_eval_tests_commands(self) -> list[str]:
+ """Build shell commands that strip the held-out eval tests from the agent's checkout.
+
+ ``/r2e_tests`` holds the evaluation tests the agent must not see;
+ ``run_tests.sh`` launches them. ``run_tests.sh`` is deleted only when it
+ references ``r2e_tests`` (substring guard) to avoid clobbering an
+ unrelated file with that name. The agent adapter runs these after
+ ``materialize``; the verifier does NOT (the nested harness needs the
+ tests back for grading).
+
+ Returns:
+ list[str]: One shell command per checkout root (``""``, ``/root``,
+ ``/testbed``) that removes the eval tests and the launcher script.
+ """
+ commands: list[str] = []
+ for root_dir in ["", "/root", "/testbed"]:
+ commands.append(
+ f"rm -rf {root_dir}/r2e_tests && "
+ f"if grep -qs r2e_tests {root_dir}/run_tests.sh; then rm -rf {root_dir}/run_tests.sh; fi"
+ )
+ return commands
+
+ async def run_eval(self, env: "AsyncSweEnvironment", task: SweTask) -> EvalArtifacts:
+ """Run evaluation for an r2e-gym task and collect its artifacts.
+
+ In opt-in flat mode, runs the instance's eval script in-sandbox and
+ defers to the flat eval path. Otherwise runs the nested
+ ``run_local_evaluation`` harness in-container: it reads the model patch
+ from the predictions file, applies it, runs the held-out tests, and
+ writes ``report.json``, which is copied to a stable path and read back
+ host-side for grading.
+
+ Args:
+ env: The active SWE environment used to execute commands in the
+ sandbox.
+ task: The SWE task supplying metadata such as setup dir, predictions
+ path, dataset path, timeout, and output dir.
+
+ Returns:
+ EvalArtifacts: The captured report text (or command output), return
+ code, whether the patch was treated as applied, and raw fields
+ including the error type and report JSON.
+ """
+ # Opt-in flat mode: run the instance's eval script in-sandbox and grade
+ # the log host-side. Default path below is the nested
+ # run_local_evaluation harness (apptainer-only).
+ if flat_eval.flat_eval_enabled(self.flat_eval, task):
+ return await flat_eval.flat_run_eval(env, task)
+
+ # The nested r2e-gym harness reads the model patch from the predictions
+ # file, applies it, runs the held-out tests, and writes ``report.json``.
+ # We build the in-container command and redirect its report to
+ # ``_REPORT_PATH``, then read it back host-side for grading.
+ setup_dir = task.metadata.get("r2egym_setup_dir", "/r2egym_setup")
+ predictions_path = task.metadata.get("predictions_path", _PREDICTIONS_PATH)
+ dataset_path = task.metadata.get("dataset_path", "/root/dataset/data.jsonl")
+ timeout = task.metadata.get("tests_timeout", 1800)
+ output_dir = task.metadata.get("eval_output_dir", "/root/eval-outputs")
+ eval_cmd = (
+ "cd /r2egym_setup/R2E-Gym && "
+ f'export UV_INSTALL_DIR="{setup_dir}/uv" && '
+ f'export UV_PYTHON_INSTALL_DIR="{setup_dir}/python" && '
+ f'export PATH="{setup_dir}/uv/bin:$PATH" && '
+ f"env -u VIRTUAL_ENV {setup_dir}/R2E-Gym/venv/bin/python "
+ "src/r2egym/agenthub/run/run_local_evaluation.py "
+ f"--predictions_path {predictions_path} "
+ f"--instance_id {task.instance_id} "
+ f"--timeout {timeout} "
+ f"--dataset {dataset_path} "
+ f"--output_dir {output_dir} && "
+ # Surface the per-instance report at a stable, well-known path.
+ f"cp {output_dir}/report.json {_REPORT_PATH}"
+ )
+ result = await env.execute(eval_cmd, cwd=task.repo_workdir, is_eval=True, timeout_s=timeout + 120)
+ report_text = ""
+ if result["returncode"] == 0:
+ report = await env.execute(f"cat {_REPORT_PATH}", cwd=task.repo_workdir, is_eval=True)
+ if report["returncode"] == 0:
+ report_text = report["output"]
+ return EvalArtifacts(
+ test_output=report_text or result["output"],
+ return_code=result["returncode"],
+ # The nested harness applies the patch itself; absent a host apply
+ # step we treat a clean eval as "applied" and let grade() mask
+ # infra failures via error_kind.
+ patch_applied=result["returncode"] == 0,
+ raw={"error_type": result.get("error_type"), "report_json": report_text},
+ )
+
+ def grade(self, task: SweTask, artifacts: EvalArtifacts) -> SweEvalReport:
+ """Grade an r2e-gym task from its evaluation artifacts.
+
+ In flat mode (from the harness flag/task opt-in or flat artifacts),
+ parses the eval-script log host-side. Otherwise masks infra failures via
+ ``error_kind`` and parses the nested harness's ``report.json``, trusting
+ its already-computed ``resolved`` verdict.
+
+ Args:
+ task: The SWE task being graded, supplying the instance id and model
+ patch.
+ artifacts: The evaluation artifacts produced by ``run_eval``,
+ including the report JSON and raw error/flat markers.
+
+ Returns:
+ SweEvalReport: The resolved/unresolved verdict with patch existence,
+ patch-applied status, per-test status, and any error kind.
+ """
+ # Flat mode: host-side parse of the eval-script log. Detected from either
+ # the harness flag/task opt-in OR the artifacts produced by flat_run_eval
+ # (so a flat run_eval is always graded flat, even on a shared instance).
+ if flat_eval.flat_eval_enabled(self.flat_eval, task) or artifacts.raw.get("flat"):
+ return flat_eval.flat_grade(task, artifacts)
+
+ # Infra failure → mask via error_kind (never scored as "unresolved").
+ if artifacts.raw.get("error_type") in {"sandbox", "timeout"}:
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ patch_applied=artifacts.patch_applied,
+ error_kind=artifacts.raw["error_type"],
+ )
+ report_text = artifacts.raw.get("report_json") or artifacts.test_output
+ try:
+ report = json.loads(report_text)
+ except (json.JSONDecodeError, TypeError):
+ # The nested harness never produced a parseable report → eval error.
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ patch_applied=artifacts.patch_applied,
+ error_kind="eval_error",
+ )
+ # report.json is keyed by instance_id (standard SWE-bench shape); fall
+ # back to the sole entry if the key was rewritten.
+ entry = report.get(task.instance_id)
+ if entry is None and len(report) == 1:
+ entry = next(iter(report.values()))
+ if not isinstance(entry, dict):
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ patch_applied=artifacts.patch_applied,
+ error_kind="eval_error",
+ )
+ # The nested harness has already computed ``resolved``; trust it.
+ resolved = bool(entry.get("resolved", False))
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ resolved=resolved,
+ patch_applied=artifacts.patch_applied,
+ patch_exists=bool(task.model_patch),
+ tests_status=entry.get("tests_status", {}),
+ )
diff --git a/responses_api_agents/swe_env/harnesses/swe_bench_ext.py b/responses_api_agents/swe_env/harnesses/swe_bench_ext.py
new file mode 100644
index 0000000000..6114d128fe
--- /dev/null
+++ b/responses_api_agents/swe_env/harnesses/swe_bench_ext.py
@@ -0,0 +1,260 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""swe-bench-ext harness: flat, host-graded reference family.
+
+Applies the model patch (and test patch) against the repository checkout, runs
+the framework test command, and grades host-side with the parser
+(:func:`responses_api_agents.swe_env.parsing.parse_and_check_tests`).
+
+Grading delegates the full per-framework logic to ``parse_and_check_tests``:
+junit-xml parsing, test-id normalization, the fuzzy matcher, the framework
+dispatch, the ``::build``/``::compile`` synthetic-PASS injection, and
+build-failed-package propagation.
+
+``resolved`` is taken from the parser's verdict (all FAIL_TO_PASS passed AND all
+PASS_TO_PASS passed). It does not depend on ``patch_applied``: the model and test
+patches are applied best-effort and grading is on the tests only.
+``patch_applied`` is still recorded for information.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from nemo_gym.sandbox import SandboxResources, SandboxSpec
+from responses_api_agents.swe_env.harness import EvalArtifacts, SweEvalReport, SweTask, SweTaskHarness
+from responses_api_agents.swe_env.parsing import (
+ get_framework_config,
+ get_test_command_with_output,
+ parse_and_check_tests,
+)
+
+
+if TYPE_CHECKING:
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+
+# Output markers the parser (parse_and_check_tests) extracts content between.
+_TEST_OUTPUT_START = "<<>>"
+_TEST_OUTPUT_END = "<<>>"
+_RESULT_FILE_START = "<<>>"
+_RESULT_FILE_END = "<<>>"
+
+
+class SweBenchExtHarness(SweTaskHarness):
+ """Flat, host-graded harness for the swe-bench-ext task family.
+
+ Runs the task's framework test command inside a single sandbox and grades the
+ captured output on the host. Works on any exec-capable sandbox provider.
+ """
+
+ name = "swe-bench-ext"
+ grade_strategy = "flat-host-grade"
+
+ def build_spec(self, task: SweTask) -> SandboxSpec:
+ """Build the sandbox specification for a task.
+
+ Args:
+ task: The SWE task describing the image, working directory, and
+ per-task metadata (timeouts, resources, provider options).
+
+ Returns:
+ SandboxSpec: The sandbox spec used to launch the task's container.
+ """
+ return SandboxSpec(
+ image=task.image,
+ workdir=task.repo_workdir,
+ ttl_s=task.metadata.get("ttl_s", 1800),
+ ready_timeout_s=task.metadata.get("ready_timeout_s", 600),
+ env={"GIT_CONFIG_GLOBAL": "/dev/null", "GIT_PAGER": "cat"},
+ metadata={
+ "instance_id": task.instance_id[:63],
+ "benchmark": task.benchmark,
+ "harness": self.name,
+ },
+ resources=SandboxResources.from_mapping(task.metadata.get("resources", {})),
+ provider_options=task.metadata.get("provider_options", {}),
+ )
+
+ def supports_provider(self, provider_name: str) -> bool:
+ """Report whether this harness supports a sandbox provider.
+
+ Being flat and host-graded, it works on any exec-capable provider.
+
+ Args:
+ provider_name: The name of the sandbox provider.
+
+ Returns:
+ bool: Always ``True``.
+ """
+ return True
+
+ async def run_eval(self, env: "AsyncSweEnvironment", task: SweTask) -> EvalArtifacts:
+ """Apply patches, run the test command, and capture the evaluation output.
+
+ Applies the model patch (and test patch) best-effort, then runs the
+ framework test command wrapped between output markers so the parser can
+ extract the structured result file or marked stdout.
+
+ Args:
+ env: The async environment used to execute commands in the sandbox.
+ task: The SWE task providing the patches, test command, and framework.
+
+ Returns:
+ EvalArtifacts: The captured test output, return code, whether the
+ model patch applied, and the execution error type if any.
+ """
+ workdir = task.repo_workdir
+ patch_applied = True
+ # Best-effort apply: a bad apply never fails the run (grading is on the
+ # tests only); we still record whether the model patch applied for info.
+ apply_flags = "--reject --recount --ignore-space-change --ignore-whitespace"
+ if task.model_patch:
+ applied = await env.execute(
+ f"git apply {apply_flags} /root/patch.diff",
+ cwd=workdir,
+ )
+ patch_applied = applied["returncode"] == 0
+ if task.test_patch:
+ await env.execute(
+ f"git apply {apply_flags} /root/test_patch.diff",
+ cwd=workdir,
+ )
+ # Wrap the command's output: add structured-output flags (--junitxml/--json)
+ # via get_test_command_with_output, run it between the markers, and dump the
+ # framework result file so parse_and_check_tests receives junit-xml (preferred)
+ # or the marked stdout.
+ #
+ # The framework is passed through verbatim. An empty framework must NOT be
+ # coerced to "pytest": for a non-pytest instance whose framework is absent, the
+ # parser's auto-detect path is what grades correctly, and the default framework
+ # config adds no flags and no result file. grade() reuses this SAME value via
+ # _resolve_framework so the two stay in lockstep.
+ framework = self._resolve_framework(task)
+ # Keep a command default ONLY when test_command is truly absent. Real
+ # swe-bench-ext rows carry their own command; the default covers rows that do
+ # not ship one without altering any row that does.
+ base_command = task.test_command or "python -m pytest -rA -q"
+ test_cmd = get_test_command_with_output(base_command, framework)
+ result_file = (get_framework_config(framework, base_command) or {}).get("result_file")
+ result = await env.execute(self._wrap_eval_command(test_cmd, result_file), cwd=workdir, is_eval=True)
+ return EvalArtifacts(
+ test_output=result["output"],
+ return_code=result["returncode"],
+ patch_applied=patch_applied,
+ raw={"error_type": result.get("error_type")},
+ )
+
+ @staticmethod
+ def _resolve_framework(task: SweTask) -> str:
+ """Return the framework value used by both ``run_eval`` and ``grade``.
+
+ Returns the task's framework verbatim. An empty or unknown value is
+ intentionally passed through unchanged: coercing it to ``"pytest"`` would
+ mis-dispatch the parser for non-pytest instances that ship no framework.
+ Centralizing this guarantees ``run_eval`` (which selects the
+ structured-output flag and result file) and ``grade`` (which parses the
+ output) agree on the framework.
+
+ Args:
+ task: The SWE task whose framework value is returned.
+
+ Returns:
+ str: The task's test framework name (possibly empty).
+ """
+ return task.test_framework
+
+ @staticmethod
+ def _wrap_eval_command(test_cmd: str, result_file: str | None) -> str:
+ """Wrap the eval command in the output markers and a result-file dump.
+
+ The parser prefers the junit/json result file (emitted between the
+ RESULT_FILE markers) and falls back to the marked stdout. The ``mkdir -p``
+ ensures ``/workspace/test-results`` exists first, since some frameworks
+ (e.g. junit/gradle, xctest) write their result file there.
+
+ Args:
+ test_cmd: The test command to run inside the markers.
+ result_file: Path or glob of the framework result file to dump, or
+ ``None`` when the framework produces no result file.
+
+ Returns:
+ str: A shell script that runs the test command and emits the marked
+ output and result-file blocks.
+ """
+ mkdir_block = "mkdir -p /workspace/test-results\n"
+ if result_file and "*" in result_file:
+ result_block = (
+ f'echo "{_RESULT_FILE_START}"\n'
+ f"for f in {result_file}; do\n"
+ f' if [ -f "$f" ]; then echo "=== FILE: $f ==="; cat "$f"; echo ""; fi\n'
+ f"done 2>/dev/null || true\n"
+ f'echo "{_RESULT_FILE_END}"\n'
+ )
+ elif result_file:
+ result_block = (
+ f'echo "{_RESULT_FILE_START}"\n'
+ f'if [ -f "{result_file}" ]; then cat "{result_file}"; fi\n'
+ f'echo "{_RESULT_FILE_END}"\n'
+ )
+ else:
+ result_block = ""
+ return f'{mkdir_block}echo "{_TEST_OUTPUT_START}"\n{test_cmd}\n{result_block}echo "{_TEST_OUTPUT_END}"\n'
+
+ def grade(self, task: SweTask, artifacts: EvalArtifacts) -> SweEvalReport:
+ """Grade captured evaluation artifacts into a report.
+
+ Infrastructure failures are masked via ``error_kind`` and never scored as
+ unresolved. Otherwise the test output is handed to ``parse_and_check_tests``
+ and ``resolved`` is taken from the parser's verdict.
+
+ Args:
+ task: The SWE task providing the expected test sets and framework.
+ artifacts: The captured test output, return code, and error type.
+
+ Returns:
+ SweEvalReport: The grading report, including ``resolved``,
+ ``patch_applied``, ``patch_exists``, and the parsed test status (or
+ ``error_kind`` on infrastructure failure).
+ """
+ # Infra failure: mask via error_kind (never scored as "unresolved").
+ if artifacts.raw.get("error_type") in {"sandbox", "timeout"}:
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ patch_applied=artifacts.patch_applied,
+ error_kind=artifacts.raw["error_type"],
+ )
+ # Delegate to the parser, passing the framework verbatim via the SAME
+ # _resolve_framework value run_eval used. An empty/unknown framework falls
+ # through to the parser's auto-detect path; coercing it to "pytest" here would
+ # mis-grade non-pytest instances.
+ test_framework = self._resolve_framework(task)
+ result = parse_and_check_tests(
+ test_output=artifacts.test_output,
+ test_framework=test_framework,
+ fail_to_pass=task.fail_to_pass,
+ pass_to_pass=task.pass_to_pass,
+ instance_id=task.instance_id,
+ )
+ # resolved is the parser's verdict (all F2P passed AND all P2P passed); it
+ # does NOT gate on patch_applied (grading is on tests only).
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ resolved=bool(result["resolved"]),
+ patch_applied=artifacts.patch_applied,
+ patch_exists=bool(task.model_patch),
+ tests_status=result,
+ )
diff --git a/responses_api_agents/swe_env/harnesses/swe_rebench.py b/responses_api_agents/swe_env/harnesses/swe_rebench.py
new file mode 100644
index 0000000000..bb8cdab250
--- /dev/null
+++ b/responses_api_agents/swe_env/harnesses/swe_rebench.py
@@ -0,0 +1,375 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""swe-rebench harness: a flat, host-graded family with a vendored log parser.
+
+This is a flat host-graded family: reset to base, apply the model patch and test
+patch, run the install/test commands, then parse the test log host-side.
+
+Two things distinguish swe-rebench:
+
+* **JAVA env** — SWE-rebench tasks need
+ ``_JAVA_OPTIONS=-Djava.net.preferIPv6Addresses=false``, surfaced via
+ ``build_spec.env`` so it is set for the whole sandbox session.
+* **Dynamic log parser** — swe-rebench has no single uniform pytest summary; the
+ correct per-test PASSED/FAILED status comes from a repo-specific parser keyed
+ by ``log_parser`` and shipped in the cloned ``SWE-rebench-V2`` repo
+ (``lib/agent/log_parsers.py`` or ``agent/log_parsers.py``). It is imported
+ dynamically, guarded by try/except.
+
+The cloned ``SWE-rebench-V2`` directory must be provisioned out-of-band. When it
+is absent or the named parser cannot be resolved, ``grade`` masks the sample via
+``error_kind`` rather than scoring a misleading ``unresolved``.
+"""
+
+from __future__ import annotations
+
+import importlib.util
+import json
+import re
+import sys
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, Callable
+
+from nemo_gym.sandbox import SandboxResources, SandboxSpec
+from responses_api_agents.swe_env.harness import EvalArtifacts, SweEvalReport, SweTask, SweTaskHarness
+
+
+if TYPE_CHECKING:
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+
+# JAVA flag required for every SWE-rebench task.
+_JAVA_OPTIONS = "-Djava.net.preferIPv6Addresses=false"
+
+# Patch-apply flags shared by the model and test patch; non-fatal
+# ``git apply --reject`` style so a failed apply still runs the tests.
+_APPLY_FLAGS = "--reject --recount --ignore-space-change --whitespace=nowarn"
+
+# Timing/duration suffixes some test runners append to node names; stripped so
+# the parser output lines up with the (already-normalized) expected node ids.
+_REBENCH_TIMING_NORMALIZE_RES = [
+ re.compile(r"\s*\[\s*\d+(?:\.\d+)?\s*(?:ms|s)\s*\]\s*$", re.IGNORECASE),
+ re.compile(r"\s+in\s+\d+(?:\.\d+)?\s+(?:msec|sec)\b", re.IGNORECASE),
+ re.compile(r"\s*\(\s*\d+(?:\.\d+)?\s*(?:ms|s)\s*\)\s*$", re.IGNORECASE),
+]
+
+
+def _normalize_test_name(name: str) -> str:
+ """Strip trailing timing annotations from a test node name.
+
+ Args:
+ name (str): The raw test node name, possibly carrying a trailing timing
+ or duration annotation.
+
+ Returns:
+ str: The node name with any timing suffix removed and surrounding
+ whitespace stripped.
+ """
+ for pattern in _REBENCH_TIMING_NORMALIZE_RES:
+ name = pattern.sub("", name)
+ return name.strip()
+
+
+def _load_rebench_log_parsers(rebench_repo_dir: Path):
+ """Dynamically import the cloned SWE-rebench-V2 ``log_parsers`` module.
+
+ Prefers ``lib/agent/log_parsers.py`` and falls back to
+ ``agent/log_parsers.py``, temporarily prepending the repo (and its ``lib``
+ directory) to ``sys.path`` so the module's intra-repo imports resolve.
+
+ Args:
+ rebench_repo_dir (Path): Path to the cloned SWE-rebench-V2 repository.
+
+ Returns:
+ ModuleType: The imported ``log_parsers`` module.
+
+ Raises:
+ FileNotFoundError: If the cloned directory has not been provisioned and
+ no ``log_parsers.py`` can be located.
+ """
+ lp_path = rebench_repo_dir / "lib" / "agent" / "log_parsers.py"
+ if not lp_path.exists():
+ lp_path = rebench_repo_dir / "agent" / "log_parsers.py"
+ if not lp_path.exists():
+ raise FileNotFoundError(
+ f"SWE-rebench-V2 log_parsers not found under {rebench_repo_dir}; "
+ "provision the clone via setup_scripts/swe_rebench.sh"
+ )
+
+ extra_paths = [str(rebench_repo_dir), str(rebench_repo_dir / "lib")]
+ added: list[str] = []
+ for p in extra_paths:
+ if p not in sys.path:
+ sys.path.insert(0, p)
+ added.append(p)
+ try:
+ spec = importlib.util.spec_from_file_location("_rebench_log_parsers", str(lp_path))
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+ finally:
+ for p in added:
+ try:
+ sys.path.remove(p)
+ except ValueError:
+ pass
+
+
+def _resolve_parser(log_parsers, log_parser_name: str) -> Callable[[str], dict[str, str]] | None:
+ """Resolve a parser callable from the loaded module.
+
+ Looks up the name in the module's ``NAME_TO_PARSER`` mapping first, then
+ falls back to a module-level attribute of the same name.
+
+ Args:
+ log_parsers: The imported ``log_parsers`` module.
+ log_parser_name (str): The name of the parser to resolve.
+
+ Returns:
+ Callable[[str], dict[str, str]] | None: The resolved parser callable, or
+ ``None`` if no parser matches the name.
+ """
+ name_to_parser = getattr(log_parsers, "NAME_TO_PARSER", {}) or {}
+ return name_to_parser.get(log_parser_name) or getattr(log_parsers, log_parser_name, None)
+
+
+def _as_list(value: Any) -> list[str]:
+ """Coerce a test-command/install/list field to a list of strings.
+
+ Accepts the value as a JSON-encoded string, a bare string, or a list. A
+ JSON-encoded string is parsed and coerced recursively; a bare string that
+ fails to parse is wrapped in a single-element list.
+
+ Args:
+ value (Any): The field value to coerce. May be ``None``, a string, a
+ list, a tuple, or any other type.
+
+ Returns:
+ list[str]: The value normalized to a list of strings. An empty list is
+ returned for ``None`` or an empty string.
+ """
+ if value is None:
+ return []
+ if isinstance(value, str):
+ text = value.strip()
+ if not text:
+ return []
+ if text[0] in "[{":
+ try:
+ parsed = json.loads(text)
+ except (ValueError, TypeError):
+ return [value]
+ return _as_list(parsed)
+ return [value]
+ if isinstance(value, (list, tuple)):
+ return [str(v) for v in value]
+ return [str(value)]
+
+
+class SweRebenchHarness(SweTaskHarness):
+ """Flat, host-graded harness for the swe-rebench benchmark family.
+
+ Applies the model and test patches, runs the install/test commands, then
+ parses the test log host-side using a repo-specific parser loaded
+ dynamically from the cloned SWE-rebench-V2 repository.
+ """
+
+ name = "swe-rebench"
+ grade_strategy = "flat-host-grade"
+
+ def build_spec(self, task: SweTask) -> SandboxSpec:
+ """Build the sandbox spec for a swe-rebench task.
+
+ Sets the git and ``_JAVA_OPTIONS`` environment variables, merges any
+ task-provided env, and forwards TTL, readiness timeout, resources, and
+ provider options from the task metadata.
+
+ Args:
+ task (SweTask): The task to build a sandbox specification for.
+
+ Returns:
+ SandboxSpec: The sandbox specification for running the task.
+ """
+ # _JAVA_OPTIONS forces IPv4 for SWE-rebench tasks.
+ env = {
+ "GIT_CONFIG_GLOBAL": "/dev/null",
+ "GIT_PAGER": "cat",
+ "_JAVA_OPTIONS": _JAVA_OPTIONS,
+ }
+ env.update(task.metadata.get("env", {}))
+ return SandboxSpec(
+ image=task.image,
+ workdir=task.repo_workdir,
+ ttl_s=task.metadata.get("ttl_s", 1800),
+ ready_timeout_s=task.metadata.get("ready_timeout_s", 600),
+ env=env,
+ metadata={
+ "instance_id": task.instance_id[:63],
+ "benchmark": task.benchmark,
+ "harness": self.name,
+ },
+ resources=SandboxResources.from_mapping(task.metadata.get("resources", {})),
+ provider_options=task.metadata.get("provider_options", {}),
+ )
+
+ def supports_provider(self, provider_name: str) -> bool:
+ """Report whether the harness supports a given sandbox provider.
+
+ Being flat and host-graded, it works on any exec-capable provider.
+
+ Args:
+ provider_name (str): The name of the sandbox provider.
+
+ Returns:
+ bool: Always ``True``.
+ """
+ return True # flat, host-graded: works on any exec-capable provider
+
+ async def run_eval(self, env: "AsyncSweEnvironment", task: SweTask) -> EvalArtifacts:
+ """Apply patches, run install and test commands, and collect artifacts.
+
+ Applies the model patch then the test patch (both best-effort), runs the
+ non-fatal install commands, then runs the test block with the eval
+ timeout. Records whether the model patch applied for informational
+ purposes only; grading does not gate on it.
+
+ Args:
+ env (AsyncSweEnvironment): The environment used to execute commands
+ inside the sandbox.
+ task (SweTask): The task being evaluated.
+
+ Returns:
+ EvalArtifacts: The captured test output, return code, model-patch
+ application status, and raw error metadata.
+ """
+ workdir = task.repo_workdir
+ install_config = task.metadata.get("install_config", {}) or {}
+ install_cmds = _as_list(install_config.get("install"))
+ test_cmds = _as_list(install_config.get("test_cmd")) or ([task.test_command] if task.test_command else [])
+
+ # Apply the model patch first, then the test patch. Both are best-effort:
+ # a failed apply still runs the tests; model-patch application is recorded
+ # for info only (grading does not gate on it).
+ patch_applied = True
+ if task.model_patch:
+ applied = await env.execute(
+ f"git apply {_APPLY_FLAGS} /root/patch.diff",
+ cwd=workdir,
+ )
+ patch_applied = applied["returncode"] == 0
+ if task.test_patch:
+ await env.execute(f"git apply {_APPLY_FLAGS} /root/test_patch.diff", cwd=workdir)
+
+ # Install commands are non-fatal; failures there should not abort the
+ # test run.
+ for cmd in install_cmds:
+ await env.execute(cmd, cwd=workdir)
+
+ test_block = "\n".join(test_cmds) if test_cmds else "python -m pytest -rA -q"
+ # Thread the eval timeout into the test exec, defaulting to 1800s so a
+ # stuck swe-rebench run is bounded. A row that explicitly carries a
+ # ``tests_timeout`` overrides the default.
+ result = await env.execute(
+ test_block,
+ cwd=workdir,
+ is_eval=True,
+ timeout_s=task.metadata.get("tests_timeout", 1800),
+ )
+ return EvalArtifacts(
+ test_output=result["output"],
+ return_code=result["returncode"],
+ patch_applied=patch_applied,
+ raw={"error_type": result.get("error_type")},
+ )
+
+ def grade(self, task: SweTask, artifacts: EvalArtifacts) -> SweEvalReport:
+ """Grade a swe-rebench task from its evaluation artifacts.
+
+ Masks infra failures (sandbox/timeout) and grading errors (missing clone,
+ unknown parser, parser crash) via ``error_kind`` rather than scoring them.
+ Otherwise parses the test output with the resolved repo-specific parser
+ and marks the task resolved when every FAIL_TO_PASS and PASS_TO_PASS test
+ is in the passed set.
+
+ Args:
+ task (SweTask): The task being graded.
+ artifacts (EvalArtifacts): The artifacts captured during evaluation.
+
+ Returns:
+ SweEvalReport: The grading report, with ``resolved`` set on success
+ or ``error_kind`` set when the sample is masked.
+ """
+ # Infra failure -> mask via error_kind (never scored as "unresolved").
+ if artifacts.raw.get("error_type") in {"sandbox", "timeout"}:
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ patch_applied=artifacts.patch_applied,
+ error_kind=artifacts.raw["error_type"],
+ )
+
+ install_config = task.metadata.get("install_config", {}) or {}
+ log_parser_name = install_config.get("log_parser", "")
+ # The cloned SWE-rebench-V2 dir is provisioned out-of-band; its absence,
+ # an unknown parser name, or a parser crash all mask the sample via
+ # ``error_kind`` rather than mis-scoring it.
+ rebench_repo_dir = task.metadata.get("rebench_repo_dir")
+ if not rebench_repo_dir:
+ return self._masked(task, artifacts, "eval_error")
+ try:
+ log_parsers = _load_rebench_log_parsers(Path(rebench_repo_dir))
+ parser = _resolve_parser(log_parsers, log_parser_name)
+ if parser is None:
+ return self._masked(task, artifacts, "eval_error")
+ results = parser(artifacts.test_output)
+ except Exception:
+ return self._masked(task, artifacts, "eval_error")
+
+ results = {_normalize_test_name(k): v for k, v in (results or {}).items()}
+ passed_set = {k for k, v in results.items() if v == "PASSED"}
+ fail_to_pass_set = {_normalize_test_name(n) for n in task.fail_to_pass}
+ pass_to_pass_set = {_normalize_test_name(n) for n in task.pass_to_pass}
+
+ # Resolution rule: every FAIL_TO_PASS and PASS_TO_PASS test must be in the
+ # passed set. Resolution is not gated on patch application, and the
+ # F2P/P2P sets are not required to be non-empty (an empty set is a subset
+ # of any set).
+ resolved = (fail_to_pass_set <= passed_set) and (pass_to_pass_set <= passed_set)
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ resolved=resolved,
+ patch_applied=artifacts.patch_applied,
+ patch_exists=bool(task.model_patch),
+ tests_status={"passed": sorted(passed_set), "all": results},
+ )
+
+ @staticmethod
+ def _masked(task: SweTask, artifacts: EvalArtifacts, kind: str) -> SweEvalReport:
+ """Build a masked report that records a grading error instead of a score.
+
+ Args:
+ task (SweTask): The task being graded.
+ artifacts (EvalArtifacts): The artifacts captured during evaluation.
+ kind (str): The error kind to record on the report.
+
+ Returns:
+ SweEvalReport: A report with ``error_kind`` set and no resolution.
+ """
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ patch_applied=artifacts.patch_applied,
+ error_kind=kind,
+ )
diff --git a/responses_api_agents/swe_env/harnesses/swebench.py b/responses_api_agents/swe_env/harnesses/swebench.py
new file mode 100644
index 0000000000..047125fd5d
--- /dev/null
+++ b/responses_api_agents/swe_env/harnesses/swebench.py
@@ -0,0 +1,385 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""swe-bench / swe-bench-multilingual harness with nested, in-container grading.
+
+A single parametrized class serves both families. Both run the upstream
+SWE-bench ``run_local_evaluation`` harness inside the sandbox (the pre-built
+venv is bind-mounted from the host setup dir), then read the harness's
+``report.json`` to decide ``resolved``.
+
+Because the nested harness shells out to its own Docker/Apptainer runtime to
+spin up the per-instance image, these families are gated to the ``apptainer``
+provider via ``supports_provider`` (fail-fast on exec-only providers).
+``run_eval`` builds and issues the in-container eval command, and ``grade``
+parses the emitted ``report.json``.
+"""
+
+from __future__ import annotations
+
+import json
+import shlex
+from typing import TYPE_CHECKING
+
+from nemo_gym.sandbox import SandboxResources, SandboxSpec
+from responses_api_agents.swe_env.harness import (
+ EvalArtifacts,
+ SweEvalReport,
+ SweTask,
+ SweTaskHarness,
+ _ensure_trailing_newline,
+)
+from responses_api_agents.swe_env.harnesses import flat_eval
+
+
+if TYPE_CHECKING:
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+
+# Where the nested harness reads predictions / dataset and writes its report.
+_DATASET_PATH = "/root/dataset/data.jsonl"
+_PREDICTIONS_PATH = "/root/predictions.jsonl"
+_REPORT_PATH = "/root/report.json"
+
+# Per-family in-container setup dir and harness subdir used to invoke the
+# upstream ``run_local_evaluation`` module. Keyed by harness/dataset name.
+# * swe-bench: harness mounted at /swebench_setup
+# * swe-bench-multilingual: harness mounted at /swebench_multilingual_setup
+_FAMILY_CONFIG: dict[str, dict[str, str]] = {
+ "swe-bench": {
+ "setup_dir": "/swebench_setup",
+ "harness_subdir": "SWE-bench",
+ },
+ "swe-bench-multilingual": {
+ "setup_dir": "/swebench_multilingual_setup",
+ "harness_subdir": "SWE-bench_Multilingual",
+ },
+}
+
+
+class SweBenchHarness(SweTaskHarness):
+ """Nested SWE-bench (and multilingual) harness.
+
+ A single class serves both registry keys; construct one instance per family
+ (``SweBenchHarness("swe-bench")`` / ``SweBenchHarness("swe-bench-multilingual")``)
+ or let ``grade`` fall back to ``task.benchmark`` for family-specific config.
+ """
+
+ grade_strategy = "nested-harness"
+
+ def __init__(self, name: str = "swe-bench", *, flat_eval: bool = False) -> None:
+ """Initialize the harness for a given swe-bench family.
+
+ Args:
+ name: The swe-bench family to serve, one of the keys in
+ ``_FAMILY_CONFIG`` (``"swe-bench"`` or ``"swe-bench-multilingual"``).
+ flat_eval: When True, the harness runs the instance's eval script
+ directly in the sandbox and parses the log host-side, allowing it
+ to run on any exec-capable provider. When False, the nested
+ in-container grading path is used.
+
+ Raises:
+ ValueError: If ``name`` is not a known swe-bench family.
+ """
+ if name not in _FAMILY_CONFIG:
+ raise ValueError(f"Unknown swe-bench family: {name!r} (expected one of {sorted(_FAMILY_CONFIG)})")
+ self.name = name
+ # Opt-in flat (host-graded) mode. When True the harness runs the
+ # instance's eval script directly in the sandbox and parses the log
+ # host-side, lifting the apptainer-only gate so it can run on
+ # docker/opensandbox. Default False keeps the nested behavior.
+ self.flat_eval = flat_eval
+ if flat_eval:
+ self.grade_strategy = "flat-host-grade"
+
+ # --- provisioning --------------------------------------------------------
+
+ def build_spec(self, task: SweTask) -> SandboxSpec:
+ """Build the sandbox spec for a task, including family-specific mounts.
+
+ Args:
+ task: The task to provision a sandbox for.
+
+ Returns:
+ A ``SandboxSpec`` describing the image, workdir, environment, and
+ provider options (including the dataset and harness venv bind mounts).
+ """
+ # Bind-mount the dataset JSONL and the host-built SWE-bench harness venv
+ # at both its canonical path and the in-container alias (uv hardcodes
+ # absolute paths). Surfaced via ``provider_options['mounts']``, the
+ # channel the apptainer provider consumes.
+ provider_options = dict(task.metadata.get("provider_options", {}))
+ provider_options.setdefault("mounts", self._family_mounts(task))
+ return SandboxSpec(
+ image=task.image,
+ workdir=task.repo_workdir,
+ ttl_s=task.metadata.get("ttl_s", 1800),
+ ready_timeout_s=task.metadata.get("ready_timeout_s", 600),
+ env={"GIT_CONFIG_GLOBAL": "/dev/null", "GIT_PAGER": "cat"},
+ metadata={
+ "instance_id": task.instance_id[:63],
+ "benchmark": task.benchmark,
+ "harness": self.name,
+ },
+ resources=SandboxResources.from_mapping(task.metadata.get("resources", {})),
+ provider_options=provider_options,
+ )
+
+ def supports_provider(self, provider_name: str) -> bool:
+ """Report whether this harness can run on the named sandbox provider.
+
+ Args:
+ provider_name: The name of the sandbox provider (e.g. ``"apptainer"``,
+ ``"docker"``).
+
+ Returns:
+ True if the provider is supported. Flat mode runs on any
+ exec-capable provider; nested mode requires ``apptainer``.
+ """
+ # Flat mode is host-graded (no nested container), so it runs on any
+ # exec-capable provider.
+ if self.flat_eval:
+ return True
+ # Nested family: the upstream harness manages its own container runtime.
+ # Reject exec-only providers (docker/fake) and require apptainer.
+ return provider_name == "apptainer"
+
+ async def materialize(self, env: "AsyncSweEnvironment", task: SweTask) -> None:
+ """Write the predictions JSONL the nested harness consumes.
+
+ Args:
+ env: The environment used to write files into the sandbox.
+ task: The task whose model patch is embedded as the prediction.
+ """
+ # The nested harness consumes a predictions JSONL keyed by instance_id
+ # rather than a bare patch.diff. Normalize the patch trailing newline
+ # before embedding it: a non-empty patch missing its trailing newline
+ # gets one appended; an empty/absent patch stays empty. The upstream
+ # ``git apply`` is newline-sensitive, so an unnormalized patch can fail
+ # to apply and silently flip ``resolved`` to False.
+ await env.write_text(_PREDICTIONS_PATH, json.dumps(self._prediction(task)) + "\n")
+
+ @staticmethod
+ def _prediction(task: SweTask) -> dict[str, str]:
+ """Build the prediction record for a task's model patch.
+
+ Args:
+ task: The task whose model patch and metadata populate the record.
+
+ Returns:
+ A dict with the instance id, model name, and (newline-normalized)
+ model patch.
+ """
+ patch = task.model_patch or ""
+ return {
+ "instance_id": task.instance_id,
+ "model_name_or_path": task.metadata.get("model_name_or_path", "nemo-gym"),
+ # Only normalize a non-empty patch; an empty patch stays "".
+ "model_patch": _ensure_trailing_newline(patch) if patch else "",
+ }
+
+ # --- server-private grading ----------------------------------------------
+
+ async def run_eval(self, env: "AsyncSweEnvironment", task: SweTask) -> EvalArtifacts:
+ """Run the SWE-bench evaluation for a task and collect its artifacts.
+
+ In flat mode, runs the instance's eval script in-sandbox and grades the
+ log host-side. Otherwise, runs the nested ``run_local_evaluation``
+ harness in-container and reads back its ``report.json``.
+
+ Args:
+ env: The environment used to execute commands in the sandbox.
+ task: The task to evaluate.
+
+ Returns:
+ An ``EvalArtifacts`` carrying the test output, return code, whether a
+ patch was applied, and the raw report JSON / error type.
+ """
+ # Opt-in flat mode: run the instance's eval script in-sandbox and grade
+ # the log host-side. Default path below is the nested
+ # run_local_evaluation harness (apptainer-only).
+ if flat_eval.flat_eval_enabled(self.flat_eval, task):
+ return await flat_eval.flat_run_eval(env, task)
+
+ cfg = self._family_config(task)
+ # A single host-setup key serves both halves of the harness so the bind
+ # source and the cd/UV/venv path can never disagree. Default to the
+ # family alias (``cfg["setup_dir"]``), which equals the in-container
+ # mount alias and the SWE-bench-Verified default; a verifier-provisioned
+ # real host dir overrides via the same key.
+ setup_dir = self._setup_dir(task)
+ harness_subdir = cfg["harness_subdir"]
+ venv_python = f"{setup_dir}/{harness_subdir}/venv/bin/python"
+ timeout = int(task.metadata.get("tests_timeout", 1800))
+ run_id = task.metadata.get("run_id", task.instance_id)
+ split = task.split or "test"
+
+ # Build the in-container eval command: run the upstream harness against
+ # the materialized predictions and redirect its report.json to a known
+ # path. The UV_* exports and PATH point uv/python at the mounted portable
+ # dirs so the pre-built venv resolves its hardcoded toolchain.
+ eval_cmd = (
+ f"cd {setup_dir}/{harness_subdir} && "
+ f'export UV_INSTALL_DIR="{setup_dir}/uv" && '
+ f'export UV_PYTHON_INSTALL_DIR="{setup_dir}/python" && '
+ f'export PATH="{setup_dir}/uv/bin:$PATH" && '
+ f"env -u VIRTUAL_ENV {shlex.quote(venv_python)} -m swebench.harness.run_local_evaluation "
+ f"--predictions_path {shlex.quote(_PREDICTIONS_PATH)} "
+ f"--instance_ids {shlex.quote(task.instance_id)} "
+ f"--timeout {timeout} "
+ f"--dataset_name {shlex.quote(_DATASET_PATH)} "
+ f"--split {shlex.quote(split)} "
+ f"--run_id {shlex.quote(str(run_id))}"
+ )
+ # The upstream harness writes logs/run_evaluation////report.json;
+ # locate it and copy to a stable path so grade() can read a single file.
+ collect_cmd = (
+ f"REPORT=$(find logs/run_evaluation/{shlex.quote(str(run_id))} -name report.json | head -n1); "
+ f'if [ -n "$REPORT" ]; then cp "$REPORT" {shlex.quote(_REPORT_PATH)}; fi'
+ )
+ # Thread the eval timeout (tests_timeout + 120s headroom) so a stuck
+ # nested harness is killed and masked via error_kind rather than hanging
+ # the verifier.
+ result = await env.execute(
+ f"{eval_cmd} && {collect_cmd}", cwd=task.repo_workdir, is_eval=True, timeout_s=timeout + 120
+ )
+
+ # Read the emitted report.json back out of the sandbox for host-side grading.
+ report_text = ""
+ if result.get("error_type") not in {"sandbox", "timeout"}:
+ cat = await env.execute(f"cat {shlex.quote(_REPORT_PATH)}", cwd=task.repo_workdir)
+ if cat["returncode"] == 0:
+ report_text = cat["output"]
+
+ return EvalArtifacts(
+ test_output=result["output"],
+ return_code=result["returncode"],
+ patch_applied=bool(task.model_patch),
+ raw={"error_type": result.get("error_type"), "report_json": report_text},
+ )
+
+ def grade(self, task: SweTask, artifacts: EvalArtifacts) -> SweEvalReport:
+ """Grade a task from its evaluation artifacts.
+
+ Args:
+ task: The task being graded.
+ artifacts: The evaluation artifacts produced by ``run_eval``.
+
+ Returns:
+ A ``SweEvalReport`` recording resolution, patch state, and any error
+ kind. Infrastructure failures are masked via ``error_kind`` rather
+ than scored as unresolved.
+ """
+ # Flat mode: host-side parse of the eval-script log. Detected from either
+ # the harness flag/task opt-in OR the artifacts produced by flat_run_eval
+ # (so a flat run_eval is always graded flat, even on a shared instance).
+ if flat_eval.flat_eval_enabled(self.flat_eval, task) or artifacts.raw.get("flat"):
+ return flat_eval.flat_grade(task, artifacts)
+
+ # Infra failure -> mask via error_kind (never scored as "unresolved").
+ if artifacts.raw.get("error_type") in {"sandbox", "timeout"}:
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ patch_applied=artifacts.patch_applied,
+ error_kind=artifacts.raw["error_type"],
+ )
+
+ report_text = artifacts.raw.get("report_json") or ""
+ resolved = False
+ try:
+ report = json.loads(report_text)
+ # Upstream harness keys report.json by instance_id.
+ entry = report.get(task.instance_id, {}) if isinstance(report, dict) else {}
+ resolved = bool(entry.get("resolved", False))
+ except (json.JSONDecodeError, TypeError, AttributeError):
+ # Missing / malformed report -> eval failure; mask rather than score 0.
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ patch_exists=bool(task.model_patch),
+ patch_applied=artifacts.patch_applied,
+ error_kind="eval_error",
+ )
+
+ return SweEvalReport(
+ instance_id=task.instance_id,
+ resolved=resolved,
+ patch_applied=artifacts.patch_applied,
+ patch_exists=bool(task.model_patch),
+ tests_status={"report": report_text},
+ )
+
+ # --- helpers -------------------------------------------------------------
+
+ def _family_config(self, task: SweTask) -> dict[str, str]:
+ """Resolve the per-family setup config for a task.
+
+ Args:
+ task: The task whose family config is resolved.
+
+ Returns:
+ The ``_FAMILY_CONFIG`` entry for this instance's name, falling back
+ to ``task.benchmark`` and then to the ``swe-bench`` default.
+ """
+ # Prefer the instance's own name; fall back to task.benchmark so a single
+ # shared instance can still serve either family.
+ name = self.name if self.name in _FAMILY_CONFIG else task.benchmark
+ return _FAMILY_CONFIG.get(name, _FAMILY_CONFIG["swe-bench"])
+
+ def _setup_dir(self, task: SweTask) -> str:
+ """Resolve the host setup dir used for both mounting and evaluation.
+
+ Args:
+ task: The task whose metadata may carry a setup dir override.
+
+ Returns:
+ The setup dir path, preferring ``setup_dir`` then ``host_setup_dir``
+ from task metadata, then the family in-container alias.
+ """
+ # A single host-setup key is consumed by both ``build_spec`` (mount
+ # source) and ``run_eval`` (cd/UV/venv path). Accept either key
+ # (``setup_dir`` first, then ``host_setup_dir``), then fall back to the
+ # family in-container alias (which is also the SWE-bench-Verified default
+ # and the canonical mount target).
+ return (
+ task.metadata.get("setup_dir")
+ or task.metadata.get("host_setup_dir")
+ or self._family_config(task)["setup_dir"]
+ )
+
+ def _family_mounts(self, task: SweTask) -> list[dict[str, str]]:
+ """Build the bind mounts for a task's dataset and harness setup dir.
+
+ Args:
+ task: The task whose dataset path and setup dir define the mounts.
+
+ Returns:
+ A list of ``{"src", "dst"}`` mount entries for the dataset and the
+ host setup dir (bound at both the alias and its canonical path).
+ """
+ cfg = self._family_config(task)
+ setup_dir = self._setup_dir(task)
+ mounts: list[dict[str, str]] = [
+ # Dataset mounted at the fixed in-container path the harness reads.
+ {"src": task.metadata.get("dataset_path", _DATASET_PATH), "dst": _DATASET_PATH},
+ ]
+ # Bind the host setup dir at both the alias and its canonical path (uv
+ # venvs hardcode absolute paths). When ``setup_dir`` already equals the
+ # alias (the default / no real host dir provisioned), the two binds
+ # collapse to one. ``run_eval`` reads the same key, so the bind source
+ # and the cd/UV/venv path can never disagree.
+ mounts.append({"src": setup_dir, "dst": cfg["setup_dir"]})
+ if setup_dir != cfg["setup_dir"]:
+ mounts.append({"src": setup_dir, "dst": setup_dir})
+ return mounts
diff --git a/responses_api_agents/swe_env/lifecycle.py b/responses_api_agents/swe_env/lifecycle.py
new file mode 100644
index 0000000000..9727a8f2e1
--- /dev/null
+++ b/responses_api_agents/swe_env/lifecycle.py
@@ -0,0 +1,65 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Sandbox lifecycle: a thin acquire context manager.
+
+``acquire_sandbox`` starts a fresh sandbox and always tears it down on exit —
+normal return, exception, or ``asyncio.CancelledError``.
+
+Per-task teardown relies on the ``finally`` block below, which covers graceful
+exit and cancellation. Backends that honor ``SandboxSpec.ttl_s`` (e.g.
+opensandbox) self-expire any sandbox orphaned by a hard crash (SIGKILL/OOM);
+``docker`` ignores ``ttl_s`` so its orphans (rare: only on un-catchable death)
+need a manual sweep.
+"""
+
+from __future__ import annotations
+
+from contextlib import asynccontextmanager
+from typing import Any, AsyncIterator, Mapping
+
+from nemo_gym.sandbox import SandboxProvider, SandboxSpec
+from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+
+@asynccontextmanager
+async def acquire_sandbox(
+ provider: Mapping[str, Any] | SandboxProvider,
+ spec: SandboxSpec,
+ *,
+ instance_id: str = "",
+) -> AsyncIterator[AsyncSweEnvironment]:
+ """Start a fresh sandbox, yield it, and always stop it on exit.
+
+ Args:
+ provider: Either a ``SandboxProvider`` instance or a mapping describing
+ the provider configuration used to create the sandbox.
+ spec: The ``SandboxSpec`` describing how to provision the sandbox.
+ instance_id: Identifier accepted for logging/telemetry; it does not
+ affect behavior.
+
+ Yields:
+ AsyncSweEnvironment: The started environment wrapping the sandbox,
+ which is cleaned up when the context manager exits.
+ """
+ env: AsyncSweEnvironment | None = None
+ try:
+ env = await AsyncSweEnvironment.start(provider, spec)
+ yield env
+ finally:
+ if env is not None:
+ try:
+ await env.cleanup()
+ except Exception:
+ pass
diff --git a/responses_api_agents/swe_env/model_endpoint.py b/responses_api_agents/swe_env/model_endpoint.py
new file mode 100644
index 0000000000..9063865767
--- /dev/null
+++ b/responses_api_agents/swe_env/model_endpoint.py
@@ -0,0 +1,114 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Provider-neutral in-sandbox model-server egress primitive.
+
+A self-driving agent (e.g. OpenHands) runs inside the sandbox and must reach the
+Gym model server. This resolves a sandbox-reachable endpoint per provider and
+injects only the minimal ``base_url``/``api_key``/``model`` via ``SandboxSpec.env``;
+it deliberately does not serialize the whole global-config dict into the sandbox.
+
+* apptainer: shares the host network namespace, so host loopback works.
+* opensandbox: a distinct network namespace, so it requires a cluster-reachable
+ Service/ingress URL. If one is not configured, egress is unavailable and the
+ caller must declare the agent apptainer-only for that provider.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from dataclasses import dataclass
+from typing import Any
+
+
+class ModelEgressUnavailable(RuntimeError):
+ """Raised when no sandbox-reachable model endpoint can be resolved for a provider."""
+
+
+@dataclass(frozen=True)
+class ModelEndpoint:
+ """A sandbox-reachable model-server endpoint.
+
+ Attributes:
+ base_url: The base URL the in-sandbox agent uses to reach the model server.
+ api_key: Optional API key for authenticating to the model server.
+ model: Optional model name to use.
+ """
+
+ base_url: str
+ api_key: str = ""
+ model: str = ""
+
+ def to_sandbox_env(self) -> dict[str, str]:
+ """Build the minimal set of environment variables to inject into the sandbox.
+
+ Returns:
+ dict[str, str]: Environment variables carrying the base URL and,
+ when set, the API key and model name. The global config dict is
+ never included.
+ """
+ env = {"OPENAI_BASE_URL": self.base_url, "NEMO_GYM_MODEL_BASE_URL": self.base_url}
+ if self.api_key:
+ env["OPENAI_API_KEY"] = self.api_key
+ if self.model:
+ env["NEMO_GYM_MODEL"] = self.model
+ return env
+
+
+def resolve(
+ provider_name: str,
+ model_server: Mapping[str, Any],
+ *,
+ host_loopback_url: str = "http://127.0.0.1:8000/v1",
+ opensandbox_service_url: str | None = None,
+) -> ModelEndpoint:
+ """Resolve a sandbox-reachable model endpoint for a sandbox provider.
+
+ Args:
+ provider_name: The sandbox provider name (e.g. ``"apptainer"``,
+ ``"opensandbox"``, ``"docker"``).
+ model_server: Mapping describing the model server, read for the
+ ``api_key``, ``model``, and ``base_url`` keys.
+ host_loopback_url: Fallback URL used when the provider shares the host
+ network namespace and no base URL is configured.
+ opensandbox_service_url: Cluster-reachable Service/ingress URL used for
+ the opensandbox provider when no other base URL is configured.
+
+ Returns:
+ ModelEndpoint: The resolved endpoint carrying the base URL, API key,
+ and model name.
+
+ Raises:
+ ModelEgressUnavailable: If the opensandbox provider cannot resolve a
+ cluster-reachable model-server URL (e.g. only loopback is available).
+ """
+ api_key = str(model_server.get("api_key", "") or "")
+ model = str(model_server.get("model", "") or "")
+ configured_base = str(model_server.get("base_url", "") or "")
+
+ if provider_name == "apptainer":
+ base_url = configured_base or host_loopback_url
+ elif provider_name == "opensandbox":
+ base_url = opensandbox_service_url or configured_base
+ if not base_url or "127.0.0.1" in base_url or "localhost" in base_url:
+ raise ModelEgressUnavailable(
+ "opensandbox needs a cluster-reachable model-server URL (k8s Service/ingress); "
+ "loopback is unreachable from the pod. Configure 'opensandbox_service_url', or "
+ "run the agent with the apptainer or docker provider instead."
+ )
+ else:
+ # docker / local: shares host network by default (host loopback reachable).
+ base_url = configured_base or host_loopback_url
+
+ return ModelEndpoint(base_url=base_url, api_key=api_key, model=model)
diff --git a/responses_api_agents/swe_env/parsing/__init__.py b/responses_api_agents/swe_env/parsing/__init__.py
new file mode 100644
index 0000000000..65d6b5050f
--- /dev/null
+++ b/responses_api_agents/swe_env/parsing/__init__.py
@@ -0,0 +1,52 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""SWE-Bench-Ext test-output parser.
+
+Provides the per-framework parsers, framework output config, and the
+resolution helper used by SWE harnesses for host-side grading. This
+``__init__`` re-exports the public symbols so callers can import them from a
+single location, e.g.::
+
+ from responses_api_agents.swe_env.parsing import (
+ parse_and_check_tests,
+ get_framework_config,
+ get_test_command_with_output,
+ )
+"""
+
+from responses_api_agents.swe_env.parsing.frameworks import (
+ FRAMEWORK_CONFIGS,
+ get_framework_config,
+ get_test_command_with_output,
+)
+from responses_api_agents.swe_env.parsing.parsing import (
+ normalize_test_id,
+ parse_test_output,
+)
+from responses_api_agents.swe_env.parsing.utils import parse_and_check_tests
+
+
+__all__ = [
+ # High-level grading entry point (F2P/P2P resolution).
+ "parse_and_check_tests",
+ # Framework output config + command augmentation.
+ "FRAMEWORK_CONFIGS",
+ "get_framework_config",
+ "get_test_command_with_output",
+ # Framework dispatcher + test-id normalization.
+ "parse_test_output",
+ "normalize_test_id",
+]
diff --git a/responses_api_agents/swe_env/parsing/frameworks.py b/responses_api_agents/swe_env/parsing/frameworks.py
new file mode 100644
index 0000000000..7de570c491
--- /dev/null
+++ b/responses_api_agents/swe_env/parsing/frameworks.py
@@ -0,0 +1,174 @@
+#!/usr/bin/env python3
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Test framework output configuration mapping."""
+
+from typing import Dict
+
+
+FRAMEWORK_CONFIGS: Dict[str, Dict] = {
+ "pytest": {
+ "output_flag": "--junitxml=/workspace/test-results/output.xml",
+ "result_file": "/workspace/test-results/output.xml",
+ },
+ "unittest": {
+ "output_flag": "--junitxml=/workspace/test-results/output.xml",
+ "result_file": "/workspace/test-results/output.xml",
+ },
+ "go": {
+ "output_flag": "-json",
+ "result_file": None,
+ },
+ "jest": {
+ "output_flag": "--json --outputFile=/workspace/test-results/output.json",
+ "result_file": "/workspace/test-results/output.json",
+ },
+ "vitest": {
+ "output_flag": "--reporter=json --outputFile=/workspace/test-results/output.json",
+ "result_file": "/workspace/test-results/output.json",
+ },
+ "mocha": {
+ "output_flag": "--reporter json --reporter-options output=/workspace/test-results/output.json",
+ "result_file": "/workspace/test-results/output.json",
+ },
+ "bun": {
+ "output_flag": None, # Bun doesn't have structured JSON output flag by default
+ "result_file": None, # Parse from stdout
+ },
+ "junit": {
+ "output_flag": None,
+ "result_file": "find:/workspace/repo:*/target/surefire-reports:TEST-*.xml",
+ },
+ "maven": {
+ "output_flag": None,
+ "result_file": "find:/workspace/repo:*/target/surefire-reports:TEST-*.xml",
+ },
+ "gtest": {
+ "output_flag": "--gtest_output=json:/workspace/test-results/output.json",
+ "result_file": "/workspace/test-results/output.json",
+ },
+ "cargo-nextest": {
+ "output_flag": None, # Profile is already in test_command
+ "result_file": None, # JUnit XML is output to repo/junit.xml by profile config
+ },
+ "ctest": {
+ "output_flag": "--output-on-failure --output-junit /workspace/test-results/output.xml",
+ "result_file": "/workspace/test-results/output.xml",
+ },
+ "xctest": {
+ # For SwiftPM with XCTest framework
+ "output_flag": "--parallel --num-workers=1 --xunit-output /workspace/test-results/output.xml",
+ "result_file": "/workspace/test-results/output.xml",
+ },
+ "testing": {
+ # For SwiftPM with new Swift Testing framework (Swift 6+)
+ "output_flag": "--disable-xctest --parallel --xunit-output /workspace/test-results/output.xml",
+ "result_file": "/workspace/test-results/output.xml",
+ },
+ "cppunit": {
+ "output_flag": None,
+ "result_file": None,
+ },
+ # Lua test frameworks - Tier 1 (Standard XML output)
+ "busted": {
+ "output_flag": "--output=junit",
+ "result_file": "/workspace/test-results/output.xml",
+ },
+ "luaunit": {
+ "output_flag": "-o junit -n /workspace/test-results/output.xml",
+ "result_file": "/workspace/test-results/output.xml",
+ },
+ # Lua test frameworks - Tier 2 (Custom parsers)
+ "telescope": {
+ "output_flag": None,
+ "result_file": None,
+ },
+ "lust": {
+ "output_flag": None,
+ "result_file": None,
+ },
+ "minitest": {
+ "output_flag": None,
+ "result_file": None,
+ },
+ "bespoke_libgeos": {
+ "output_flag": None,
+ "result_file": None,
+ },
+ # TAP (Test Anything Protocol) - used by tape, node-tap
+ "tap": {
+ "output_flag": None, # TAP outputs to stdout
+ "result_file": None, # Parse from stdout
+ },
+ "tape": {
+ "output_flag": None, # tape outputs TAP to stdout
+ "result_file": None, # Parse from stdout
+ },
+ # Hardhat (Solidity) - uses Mocha under the hood
+ "hardhat": {
+ "output_flag": None, # Uses Mocha console reporter by default
+ "result_file": None, # Parse from stdout
+ },
+}
+
+
+def get_framework_config(framework: str, test_command: str = "") -> Dict:
+ """Get configuration for a test framework.
+
+ Args:
+ framework: Test framework name
+ test_command: The test command (optional, used to detect Gradle vs Maven)
+ """
+ config = FRAMEWORK_CONFIGS.get(
+ framework,
+ {
+ "output_flag": None,
+ "result_file": None,
+ },
+ )
+
+ # Special handling for JUnit: detect Gradle vs Maven from command
+ if framework == "junit" and test_command:
+ if "gradlew" in test_command or "gradle " in test_command:
+ # Gradle uses different output location than Maven
+ # Use */TEST-*.xml to match both standard Gradle (test/) and Android (testDebugUnitTest/)
+ config = {
+ "output_flag": None,
+ "result_file": "find:/workspace/repo:*/build/test-results*:TEST-*.xml",
+ }
+
+ # Special handling for xctest: detect Swift Testing vs XCTest from command
+ # When --disable-xctest is used, the task is using Swift Testing, not XCTest
+ # Use the 'testing' framework config to avoid adding XCTest-only flags like --num-workers
+ if framework == "xctest" and test_command:
+ if "--disable-xctest" in test_command:
+ config = FRAMEWORK_CONFIGS.get("testing", config)
+
+ return config
+
+
+def get_test_command_with_output(base_command: str, framework: str) -> str:
+ """
+ Add structured output flags to test command.
+
+ Returns: command_with_output_flags
+ """
+ config = get_framework_config(framework, base_command)
+ output_flag = config.get("output_flag")
+
+ enhanced = f"{base_command} {output_flag}" if output_flag else base_command
+
+ return enhanced
diff --git a/responses_api_agents/swe_env/parsing/parsing.py b/responses_api_agents/swe_env/parsing/parsing.py
new file mode 100644
index 0000000000..800586adad
--- /dev/null
+++ b/responses_api_agents/swe_env/parsing/parsing.py
@@ -0,0 +1,1606 @@
+#!/usr/bin/env python3
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Test parsing utilities for build.py.
+
+Helper functions for:
+- Separating test and gold patches
+- Parsing JUnit XML and JSON test outputs
+"""
+
+import json
+import re
+import xml.etree.ElementTree as ET
+from pathlib import Path
+from typing import Dict, Optional, Tuple
+
+
+def read_patch(path: Path, skip_binary: bool = False) -> str:
+ """
+ Read the text content of a patch file optionally skipping binary files
+ """
+ parts = split_patch(path, skip_binary=skip_binary)
+ return "".join([diff for _, diff in parts])
+
+
+def split_patch(patch_path: Path, skip_binary: bool = False) -> list[Tuple[str, str]]:
+ """
+ Read a patch and partition by file.
+
+ Args:
+ patch_path (Path) - The patch file to split
+ skip_binary (bool) - Whether to exclude binary files
+
+ Returns: List of (filename, patch content) tuples
+ """
+ content = patch_path.read_text()
+ parts = []
+
+ # Split by file changes (each starts with "diff --git")
+ file_diffs = re.split(r"(diff --git.*?)(?=diff --git|\Z)", content, flags=re.DOTALL)
+
+ for i in range(0, len(file_diffs), 2):
+ if i + 1 >= len(file_diffs):
+ continue
+
+ header = file_diffs[i]
+ content = file_diffs[i + 1]
+ full_diff = header + content
+
+ # Extract filename from diff header
+ file_match = re.search(r"diff --git a/(.*?) b/", full_diff)
+ if not file_match:
+ continue
+
+ filepath = file_match.group(1)
+
+ if skip_binary:
+ binary_match = re.search(r"^GIT binary patch$", full_diff, flags=re.MULTILINE)
+ if binary_match:
+ continue
+
+ parts.append((filepath, full_diff))
+
+ return parts
+
+
+def _parse_embedded_test_results(text_output: str, test_prefix: str = "") -> Dict[str, str]:
+ """Parse embedded test results from system-out text.
+
+ This handles cases like wolfssl where a single ctest testcase runs many individual tests
+ and outputs them in a specific format within .
+
+ Expected formats:
+ - " 1: test_name : passed ( 0.00016)"
+ - " 2: test_name : failed ( 0.00016)"
+ - " 3: test_name : skipped"
+ - "HMAC-MD5 test passed!"
+ - "RSA test failed!"
+
+ Args:
+ text_output: The text content from
+ test_prefix: Prefix to add to test names (usually the testcase name)
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ """
+ results = {}
+
+ # Pattern 1: Numbered test format (wolfssl API tests)
+ # Format: " 1: test_name : passed ( 0.00016)"
+ numbered_pattern = re.compile(
+ r"^\s*\d+:\s+([^\s:]+(?:\s+[^\s:]+)*?)\s*:\s*(passed|failed|skipped)", re.MULTILINE | re.IGNORECASE
+ )
+
+ for match in numbered_pattern.finditer(text_output):
+ test_name = match.group(1).strip()
+ status = match.group(2).lower()
+
+ # Build test ID with prefix
+ if test_prefix:
+ test_id = f"{test_prefix}::{test_name}"
+ else:
+ test_id = test_name
+
+ if status == "passed":
+ results[test_id] = "PASSED"
+ elif status == "failed":
+ results[test_id] = "FAILED"
+ elif status == "skipped":
+ results[test_id] = "SKIPPED"
+
+ # Pattern 2: Unit test format (wolfssl unit tests)
+ # Format: "HMAC-MD5 test passed!"
+ # Only match lines that don't contain '---' (separator lines)
+ # Use [ \t] instead of \s to avoid matching newlines
+ unit_pattern = re.compile(
+ r"^([A-Za-z0-9_\-/]+(?:[ \t]+[A-Za-z0-9_\-/]+){0,5}?)[ \t]+test[ \t]+(passed|failed)!",
+ re.MULTILINE | re.IGNORECASE,
+ )
+
+ for match in unit_pattern.finditer(text_output):
+ test_name = match.group(1).strip()
+ status = match.group(2).lower()
+
+ # Skip if the test name contains special characters indicating it's not a real test
+ if "---" in test_name or len(test_name) > 50:
+ continue
+
+ # Build test ID with prefix
+ if test_prefix:
+ test_id = f"{test_prefix}::{test_name}"
+ else:
+ test_id = test_name
+
+ if status == "passed":
+ results[test_id] = "PASSED"
+ elif status == "failed":
+ results[test_id] = "FAILED"
+
+ # Pattern 3: FAILURES section (wolfssl API tests)
+ # Format: "FAILURES:\n 892: test_wolfSSL_CTX_load_verify_locations"
+ failures_section = re.search(r"FAILURES:\s*\n(.*?)(?:\n\s*End|$)", text_output, re.DOTALL)
+ if failures_section:
+ failure_pattern = re.compile(r"^\s*\d+:\s+([^\s:]+(?:\s+[^\s:]+)*)", re.MULTILINE)
+ for match in failure_pattern.finditer(failures_section.group(1)):
+ test_name = match.group(1).strip()
+ if test_prefix:
+ test_id = f"{test_prefix}::{test_name}"
+ else:
+ test_id = test_name
+ # Mark as failed (this overrides any previous 'passed' if it exists)
+ results[test_id] = "FAILED"
+
+ return results
+
+
+def parse_junit_xml(xml_content: str) -> Dict[str, str]:
+ """Parse JUnit XML to extract test results.
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+ Even if output contains errors (import errors, syntax errors, etc.), if we find valid
+ XML test results, we parse and return them. We only return None if we're certain
+ the framework didn't run (no test results + error indicators).
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If test framework failed to run (not the same as tests failing)
+ """
+ results = {}
+ found_any_xml = False
+
+ # PRIORITY 1 & 2: Try to parse XML documents (pure or mixed with other output)
+ # Handle multiple concatenated XML documents (from multiple test result files)
+ # Split by 0:
+ doc = "", xml_start)
+ if xml_end > xml_start:
+ xml_extracted = doc[xml_start : xml_end + len("")]
+ else:
+ xml_end = doc.find("", xml_start)
+ if xml_end > xml_start:
+ xml_extracted = doc[xml_start : xml_end + len("")]
+ else:
+ continue
+
+ try:
+ tree = ET.fromstring(xml_extracted)
+ found_any_xml = True
+ except ET.ParseError:
+ continue
+
+ # Parse all testcases from this document
+ for testcase in tree.iter("testcase"):
+ classname = testcase.get("classname", "")
+ name = testcase.get("name", "")
+ test_id = f"{classname}::{name}" if classname else name
+
+ # Check if this testcase has system-out with embedded test results
+ # This handles cases like wolfssl where a single ctest executable runs many tests
+ system_out = testcase.find("system-out")
+ embedded_results = {}
+ if system_out is not None and system_out.text:
+ embedded_results = _parse_embedded_test_results(system_out.text, classname or name)
+
+ if embedded_results:
+ # If we found embedded test results, use those instead of the testcase status
+ results.update(embedded_results)
+ elif testcase.find("failure") is not None or testcase.find("error") is not None:
+ results[test_id] = "FAILED"
+ elif testcase.find("skipped") is not None:
+ results[test_id] = "SKIPPED"
+ else:
+ results[test_id] = "PASSED"
+
+ # PRIORITY 3: If we found NO valid XML and NO results, check for error indicators
+ # Only return None if we're certain the framework failed to run
+ if not found_any_xml and not results:
+ error_indicators = [
+ "ERROR: ", # Generic error marker
+ "ImportError:", # Python import errors
+ "ModuleNotFoundError:", # Python module errors
+ "SyntaxError:", # Python syntax errors
+ "FAILED ", # Framework failure markers
+ "INTERNALERROR", # pytest internal errors
+ "collection errors", # pytest collection errors
+ "error: ", # Generic error (C++, Swift, etc.)
+ "fatal error:", # Fatal compilation errors
+ "cannot find symbol", # Java compilation errors
+ "error: build had", # Swift build errors (xctest)
+ "error: terminated", # Swift process crashes (xctest)
+ ]
+ has_errors = any(indicator in xml_content for indicator in error_indicators)
+ # Return None ONLY if: no XML found AND errors present
+ # Return empty dict if: no XML found AND no errors (rare but valid)
+ return None if has_errors else results
+
+ return results
+
+
+def parse_go_json(json_output: str) -> Dict[str, str]:
+ """Parse Go test -json output (newline-delimited JSON).
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+ Even if output contains errors (module errors, build errors, etc.), if we find valid
+ test results JSON, we parse and return it. We only return None if we're certain
+ the tests didn't run (no test results + error indicators).
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If Go tests failed to run (not the same as tests failing)
+ """
+ results = {}
+ has_valid_json = False
+
+ # PRIORITY 1: Try to parse newline-delimited JSON (valid test output)
+ for line in json_output.strip().split("\n"):
+ if not line.strip():
+ continue
+ try:
+ event = json.loads(line)
+ has_valid_json = True # Found at least one valid JSON line
+ action = event.get("Action")
+
+ # Handle test-level events
+ if "Test" in event and action in ["pass", "fail", "skip"]:
+ test_name = event.get("Test", "")
+ if test_name:
+ package = event.get("Package", "")
+ test_id = f"{package}::{test_name}" if package else test_name
+
+ if action == "pass":
+ results[test_id] = "PASSED"
+ elif action == "fail":
+ results[test_id] = "FAILED"
+ elif action == "skip":
+ results[test_id] = "SKIPPED"
+
+ # Handle package-level failures (no Test field)
+ elif "Package" in event and "Test" not in event and action == "fail":
+ package = event.get("Package", "")
+ test_id = f"{package}::package"
+ results[test_id] = "FAILED"
+
+ except json.JSONDecodeError:
+ # PRIORITY 2: Handle plaintext build failures (legitimate failures)
+ # When tests can't compile/build, Go outputs plaintext "FAIL package [build failed]"
+ # This is a legitimate test failure, not a parsing error
+ build_fail_match = re.match(r"^FAIL\s+(\S+)\s+\[build failed\]", line)
+ if build_fail_match:
+ package_name = build_fail_match.group(1)
+ results[package_name] = "FAILED"
+ has_valid_json = True # Count build failures as valid results
+
+ # PRIORITY 3: If we found NO valid JSON and NO build failures, check for error indicators
+ if not has_valid_json and not results:
+ error_indicators = [
+ "go: cannot find main module", # Module not found
+ "can't load package", # Package loading errors
+ "pattern matches no packages", # No matching packages
+ "build constraints exclude all Go files", # Build constraints error
+ ]
+ has_errors = any(indicator in json_output for indicator in error_indicators)
+ # Return None ONLY if: no JSON found AND errors present
+ # Return empty dict if: no JSON found AND no errors (rare but valid)
+ return None if has_errors else results
+
+ return results
+
+
+def parse_jest_vitest_json(json_output: str) -> Dict[str, str]:
+ """Parse Jest/Vitest JSON output.
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+ Even if output contains errors (TypeScript, npm, etc.), if we find valid
+ test results JSON, we parse and return it. We only return None if we're
+ certain the framework didn't run (no test results + error indicators).
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If Jest itself failed to run (not the same as tests failing)
+ """
+ results = {}
+
+ # PRIORITY 1: Try to parse as pure JSON (test results take precedence)
+ try:
+ data = json.loads(json_output.strip())
+ # If we got JSON, check if it has test results (even if errors exist elsewhere in output)
+ except json.JSONDecodeError:
+ # PRIORITY 2: Search for JSON markers in mixed output
+ # Even with errors in output, tests might have run and produced JSON
+ json_start = json_output.find('{"numFailed') # Jest format
+ if json_start == -1:
+ json_start = json_output.find('{"numTotalTest') # Vitest format
+ if json_start == -1:
+ json_start = json_output.find('{"test') # Alternative format
+ if json_start == -1:
+ # PRIORITY 3: No JSON found - NOW check if there are error indicators
+ # Only return None if we're sure tests didn't run (no results + errors present)
+ # NOTE: error_indicators are a LAST RESORT - we prefer finding test results
+ error_indicators = [
+ "error TS", # TypeScript compilation errors (e.g., error TS2307:)
+ "ELIFECYCLE", # npm script failures
+ "npm ERR!", # npm errors
+ "Error: Cannot find module", # Module loading errors (like Mocha)
+ "SyntaxError:", # JavaScript/TypeScript syntax errors
+ "Test suite failed to run", # Jest-specific: tests couldn't be loaded
+ "FAIL ", # Jest failure marker without JSON
+ ]
+ has_errors = any(indicator in json_output for indicator in error_indicators)
+ # Return None ONLY if: no JSON found AND errors present
+ # Return empty dict if: no JSON found AND no errors (rare but valid)
+ return None if has_errors else results
+
+ # Try to extract JSON from mixed output
+ decoder = json.JSONDecoder()
+ try:
+ data, _ = decoder.raw_decode(json_output[json_start:])
+ except json.JSONDecodeError:
+ # Could not parse JSON even after finding marker
+ return None
+
+ # At this point, we have successfully parsed JSON
+ # Check if this is Jest's error response format (Jest itself failed, not the tests)
+ # Format: {"error": {"code": 2, "summary": "", "detail": ""}}
+ # This is a structured error response, NOT test results
+ if "error" in data and "code" in data.get("error", {}):
+ # This is an error response from Jest itself, not test results
+ return None
+
+ # Check if we have the expected test results structure
+ # If we have testResults, parse it even if tests failed - those are legitimate test results
+ # Parse test results
+ if "testResults" in data:
+ for test_result in data.get("testResults", []):
+ file_path = test_result.get("name", "")
+ suite_status = test_result.get("status", "")
+ assertions = test_result.get("assertionResults", [])
+
+ # Handle suite-level failures (no assertions ran)
+ if suite_status == "failed" and len(assertions) == 0:
+ test_id = f"{file_path}::suite"
+ results[test_id] = "FAILED"
+ continue
+
+ # Handle individual test assertions
+ for assertion in assertions:
+ full_name = assertion.get("fullName", "")
+ title = assertion.get("title", "")
+ status = assertion.get("status", "")
+ test_id = f"{file_path}::{full_name}" if full_name else f"{file_path}::{title}"
+
+ if status == "passed":
+ results[test_id] = "PASSED"
+ elif status == "failed":
+ results[test_id] = "FAILED"
+ elif status in ["pending", "skipped"]:
+ results[test_id] = "SKIPPED"
+
+ # If we successfully parsed JSON but found no testResults, that's unexpected
+ # Return None to indicate this isn't valid test output
+ # (Valid Jest output should have testResults array, even if empty)
+ if "testResults" not in data:
+ return None
+
+ return results
+
+
+def parse_mocha_json(json_output: str) -> Optional[Dict[str, str]]:
+ """Parse Mocha JSON output.
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+ Even if output contains errors (module errors, syntax errors, etc.), if we find valid
+ test results JSON, we parse and return it. We only return None if we're certain
+ the framework didn't run (no test results + error indicators).
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If Mocha itself failed to run (not the same as tests failing)
+ """
+ results = {}
+
+ # PRIORITY 1: Try to parse as pure JSON (test results take precedence)
+ try:
+ data = json.loads(json_output.strip())
+ # Validate this is Mocha JSON by checking for 'stats' key
+ if "stats" not in data:
+ data = None
+ except json.JSONDecodeError:
+ data = None
+
+ # PRIORITY 2: If direct parse failed, search for JSON in mixed output
+ if data is None:
+ # Look for stats key in JSON
+ stats_pos = json_output.find('"stats"')
+ if stats_pos == -1:
+ # PRIORITY 3: No JSON found - NOW check if there are error indicators
+ error_indicators = [
+ "Error: Cannot find module", # Module loading errors
+ "SyntaxError:", # JavaScript syntax errors
+ "TypeError:", # Type errors
+ "ReferenceError:", # Reference errors
+ "No test files found", # Mocha-specific: no tests found
+ ]
+ has_errors = any(indicator in json_output for indicator in error_indicators)
+ # Return None ONLY if: no JSON found AND errors present
+ # Return empty dict if: no JSON found AND no errors (rare but valid)
+ return None if has_errors else results
+
+ # Find the opening brace before "stats"
+ json_start = json_output.rfind("{", 0, stats_pos)
+ if json_start == -1:
+ return None
+
+ # Try parsing from this position
+ json_portion = json_output[json_start:]
+
+ # Use json.JSONDecoder to find where the object ends
+ decoder = json.JSONDecoder()
+ try:
+ data, _ = decoder.raw_decode(json_portion)
+ except json.JSONDecodeError:
+ return None
+
+ # Validate extracted JSON has 'stats'
+ if "stats" not in data:
+ return None
+
+ # At this point, we have valid Mocha JSON with 'stats'
+ # Parse test results even if some tests failed - those are legitimate results
+
+ # Process passed tests
+ for test in data.get("passes", []):
+ file_path = test.get("file", "")
+ full_title = test.get("fullTitle", "")
+ test_id = f"{file_path}::{full_title}" if full_title else file_path
+ results[test_id] = "PASSED"
+
+ # Process failed tests
+ for test in data.get("failures", []):
+ file_path = test.get("file", "")
+ full_title = test.get("fullTitle", "")
+ test_id = f"{file_path}::{full_title}" if full_title else file_path
+ results[test_id] = "FAILED"
+
+ # Process pending/skipped tests
+ for test in data.get("pending", []):
+ file_path = test.get("file", "")
+ full_title = test.get("fullTitle", "")
+ test_id = f"{file_path}::{full_title}" if full_title else file_path
+ results[test_id] = "SKIPPED"
+
+ return results
+
+
+def parse_gtest_json(json_output: str) -> Dict[str, str]:
+ """Parse Google Test JSON output.
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+ Even if output contains errors (compilation errors, linking errors, etc.), if we find valid
+ test results JSON, we parse and return it. We only return None if we're certain
+ the tests didn't run (no test results + error indicators).
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If GTest itself failed to run (not the same as tests failing)
+ """
+ results = {}
+
+ # PRIORITY 1: Try to parse as pure JSON (test results take precedence)
+ try:
+ data = json.loads(json_output.strip())
+ # Validate this is GTest JSON by checking for 'testsuites' key
+ if "testsuites" not in data:
+ data = None
+ except json.JSONDecodeError:
+ data = None
+
+ # PRIORITY 2: If direct parse failed, search for JSON in mixed output
+ if data is None:
+ # Try to find JSON in mixed output
+ json_start = json_output.find('{"testsuites"')
+ if json_start == -1:
+ json_start = json_output.find('{\n "testsuites"')
+ if json_start == -1:
+ # PRIORITY 3: No JSON found - NOW check if there are error indicators
+ error_indicators = [
+ "error:", # C++ compilation errors
+ "undefined reference to", # Linking errors
+ "fatal error:", # Fatal compilation errors
+ "cannot find -l", # Linking library errors (e.g., "cannot find -lgtest")
+ ": No such file or directory", # File not found errors
+ ]
+ has_errors = any(indicator in json_output for indicator in error_indicators)
+ # Return None ONLY if: no JSON found AND errors present
+ # Return empty dict if: no JSON found AND no errors (rare but valid)
+ return None if has_errors else results
+
+ # Extract JSON object
+ json_portion = json_output[json_start:]
+ decoder = json.JSONDecoder()
+ try:
+ data, _ = decoder.raw_decode(json_portion)
+ except json.JSONDecodeError:
+ return None
+
+ # Validate extracted JSON has 'testsuites'
+ if "testsuites" not in data:
+ return None
+
+ # At this point, we have valid GTest JSON with 'testsuites'
+ # Parse test results even if some tests failed - those are legitimate results
+
+ # Parse test results from testsuites
+ testsuites = data.get("testsuites", [])
+ if not isinstance(testsuites, list):
+ testsuites = [testsuites] if isinstance(testsuites, dict) else []
+
+ for testsuite in testsuites:
+ suite_name = testsuite.get("name", "")
+
+ # Handle both 'testsuite' (array) and direct test cases
+ test_cases = testsuite.get("testsuite", [])
+ if not test_cases:
+ test_cases = testsuite.get("tests", [])
+
+ for test_case in test_cases:
+ test_name = test_case.get("name", "")
+ classname = test_case.get("classname", suite_name)
+
+ # Build test ID in format: SuiteName::TestName
+ test_id = f"{classname}::{test_name}" if classname else test_name
+
+ # Determine test status
+ status = test_case.get("status", "RUN")
+ result = test_case.get("result", "COMPLETED")
+
+ # Check for failures
+ failures = test_case.get("failures", [])
+ if failures and len(failures) > 0:
+ results[test_id] = "FAILED"
+ elif status == "NOTRUN" or result == "SKIPPED":
+ results[test_id] = "SKIPPED"
+ elif result == "COMPLETED" or status == "RUN":
+ results[test_id] = "PASSED"
+ else:
+ results[test_id] = "FAILED"
+
+ return results
+
+
+def parse_maven_text_output(text_output: str) -> Dict[str, str]:
+ """Parse Maven text output for test results."""
+ results = {}
+
+ # Look for test summary lines like:
+ # Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
+ summary_pattern = r"Tests run: (\d+),\s*Failures: (\d+),\s*Errors: (\d+),\s*Skipped: (\d+)"
+
+ # Check for compilation errors - if tests can't compile, mark them as failed
+ compilation_error_pattern = r"\[ERROR\].*?testCompile.*?Compilation failure"
+ if re.search(compilation_error_pattern, text_output, re.DOTALL | re.IGNORECASE):
+ # Find test files mentioned in compilation errors
+ test_file_pattern = r"/workspace/repo/[^/]+/src/test/java/([\w/]+)\.java"
+ for match in re.finditer(test_file_pattern, text_output):
+ test_class = match.group(1).replace("/", ".")
+ # Mark as failed due to compilation
+ results[f"{test_class}::compile"] = "FAILED"
+ # If we found compilation errors, return early
+ if results:
+ return results
+
+ # Check for BUILD FAILURE
+ if "BUILD FAILURE" in text_output:
+ # If build failed and we haven't found specific test failures, mark as generic failure
+ if not results:
+ results["maven::build"] = "FAILED"
+ return results
+
+ # Parse test run summaries per module
+ lines = text_output.split("\n")
+ current_module = None
+
+ for line in lines:
+ # Track which module we're in
+ if "Building" in line and "[" in line and "]" in line:
+ # Extract module name from lines like "[INFO] Building Docs Web 1.12-SNAPSHOT [4/4]"
+ parts = line.split("Building")
+ if len(parts) > 1:
+ module_parts = parts[1].strip().split()
+ if len(module_parts) > 0:
+ current_module = module_parts[0]
+
+ # Look for test summary
+ summary_match = re.search(summary_pattern, line)
+ if summary_match:
+ total = int(summary_match.group(1))
+ failures = int(summary_match.group(2))
+ errors = int(summary_match.group(3))
+ skipped = int(summary_match.group(4))
+
+ if total > 0:
+ # We have test counts but might not have individual test names
+ # Generate generic test IDs based on the current module
+ module_name = current_module or "unknown"
+ passed = total - failures - errors - skipped
+
+ for j in range(passed):
+ results[f"{module_name}::test_{j + 1}"] = "PASSED"
+ for j in range(failures + errors):
+ results[f"{module_name}::test_failed_{j + 1}"] = "FAILED"
+ for j in range(skipped):
+ results[f"{module_name}::test_skipped_{j + 1}"] = "SKIPPED"
+ return results
+
+
+def parse_cargo_nextest(output: str) -> Dict[str, str]:
+ """Parse cargo-nextest text output.
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+ Even if output contains errors (warnings, etc.), if we find valid test results,
+ we parse and return them. We only return None if we're certain the tests didn't
+ run (no test results + error indicators).
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If cargo-nextest failed to run (not the same as tests failing)
+ """
+ results = {}
+
+ # PRIORITY 1: Parse individual test result lines
+ # Format: PASS [ 1.588s] rusty::tests integration::linking::test_name
+ # FAIL [ 5.845s] rusty codegen::tests::parameters_tests::test_name
+ test_line_pattern = re.compile(r"^\s*(PASS|FAIL|SIGKILL|SKIP)\s+\[.*?\]\s+(.+)$", re.MULTILINE)
+
+ for match in test_line_pattern.finditer(output):
+ status = match.group(1)
+ test_name = match.group(2).strip()
+
+ if status == "PASS":
+ results[test_name] = "PASSED"
+ elif status in ("FAIL", "SIGKILL"):
+ results[test_name] = "FAILED"
+ elif status == "SKIP":
+ results[test_name] = "SKIPPED"
+
+ # PRIORITY 2: If we found NO test results, check for error indicators
+ # Only return None if we're certain tests didn't run (compilation/linking errors)
+ if not results:
+ error_indicators = [
+ "error[E", # Rust compiler errors (e.g., error[E0425])
+ "error: could not compile", # Cargo compilation errors
+ "error: linking with", # Linking errors
+ "error: aborting due to", # Compilation aborted
+ ]
+ has_errors = any(indicator in output for indicator in error_indicators)
+ # Return None ONLY if: no results found AND errors present
+ # Return empty dict if: no results found AND no errors (rare but valid - no tests in project)
+ return None if has_errors else results
+
+ return results
+
+
+def parse_bun_text(text_output: str) -> Dict[str, str]:
+ """
+ Parse Bun test framework output.
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+ Even if output contains errors (TypeScript, compilation, etc.), if we find valid
+ test results, we parse and return them. We only return None if we're
+ certain Bun didn't run (no test results + error indicators).
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If Bun itself failed to run (not the same as tests failing)
+ """
+ results = {}
+ current_file = None
+ current_describe = None
+
+ # PRIORITY 1: Try to parse test results (✓ and ✗ symbols)
+ for line in text_output.split("\n"):
+ # Track current file (lines ending with .ts: or .js:)
+ if re.match(r"^[^\s].*\.(ts|js|tsx|jsx):?\s*$", line.strip()):
+ current_file = line.strip().rstrip(":")
+ current_describe = None
+ continue
+
+ # Track describe blocks (indented text followed by colon, but not test results)
+ describe_match = re.match(r"^\s+([^✓✗\n]+):\s*$", line)
+ if describe_match:
+ current_describe = describe_match.group(1).strip()
+ continue
+
+ # Remove ANSI color codes
+ clean_line = re.sub(r"\x1b\[[0-9;]*m", "", line)
+
+ # Match passed tests: ✓ test_name [time]
+ pass_match = re.match(r"^\s*✓\s+(.+?)(?:\s+\[[\d.]+m?s\])?\s*$", clean_line)
+ if pass_match:
+ test_name = pass_match.group(1).strip()
+ # Build test ID with file, describe block, and test name
+ test_id = test_name
+ if current_file:
+ test_id = f"{current_file}::{test_name}"
+ if current_describe:
+ test_id = f"{current_file}::{current_describe} > {test_name}"
+ results[test_id] = "PASSED"
+ continue
+
+ # Match failed tests: ✗ test_name [time]
+ fail_match = re.match(r"^\s*✗\s+(.+?)(?:\s+\[[\d.]+m?s\])?\s*$", clean_line)
+ if fail_match:
+ test_name = fail_match.group(1).strip()
+ # Build test ID with file, describe block, and test name
+ test_id = test_name
+ if current_file:
+ test_id = f"{current_file}::{test_name}"
+ if current_describe:
+ test_id = f"{current_file}::{current_describe} > {test_name}"
+ results[test_id] = "FAILED"
+ continue
+
+ # Alternative format: FAIL filepath > describe > test_name
+ alt_fail_match = re.match(r"^\s*FAIL\s+(.+?)\s+>\s+(.+?)\s*$", clean_line)
+ if alt_fail_match:
+ file_path = alt_fail_match.group(1).strip()
+ test_path = alt_fail_match.group(2).strip()
+ test_id = f"{file_path}::{test_path}"
+ results[test_id] = "FAILED"
+ continue
+
+ # PRIORITY 2: If no individual test results found, try parsing summary
+ if not results:
+ # Look for summary like "5 pass, 2 fail" or "X passing (Yms)"
+ summary_match = re.search(r"(\d+)\s+pass(?:ing|ed)?.*?(\d+)\s+fail(?:ing|ed)?", text_output.lower())
+ if summary_match:
+ passed = int(summary_match.group(1))
+ failed = int(summary_match.group(2))
+
+ # Generate generic test IDs
+ for i in range(passed):
+ results[f"test_{i + 1}"] = "PASSED"
+ for i in range(failed):
+ results[f"test_failed_{i + 1}"] = "FAILED"
+
+ # PRIORITY 3: No test results found - NOW check if there are error indicators
+ # Only return None if we're sure tests didn't run (no results + errors present)
+ # NOTE: error_indicators are a LAST RESORT - we prefer finding test results
+ if not results:
+ error_indicators = [
+ "error TS", # TypeScript compilation errors (e.g., error TS2307:)
+ "Error: Cannot find module", # Module loading errors
+ "SyntaxError:", # JavaScript/TypeScript syntax errors
+ "error: ", # Generic Bun errors (lowercase 'error:')
+ "Error:", # Generic errors
+ "ModuleNotFoundError", # Module not found
+ "bun: command not found", # Bun not installed
+ "panicked at", # Bun runtime panics
+ "Segmentation fault", # Critical runtime errors
+ ]
+ has_errors = any(indicator in text_output for indicator in error_indicators)
+ # Return None ONLY if: no test results found AND errors present
+ # Return empty dict if: no test results found AND no errors (rare but valid)
+ return None if has_errors else results
+
+ return results
+
+
+def parse_cppunit_text(text_output: str) -> Dict[str, str]:
+ """Parse CppUnit text output for test results.
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+ Even if output contains errors (warnings, etc.), if we find valid test results,
+ we parse and return them. We only return None if we're certain the tests didn't
+ run (no test results + error indicators).
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If CppUnit failed to run (not the same as tests failing)
+ """
+ results = {}
+
+ # PRIORITY 1: Parse individual test result lines
+ # Format: TestClassName::testMethodName : OK
+ # TestClassName::testMethodName : FAIL
+ test_line_pattern = re.compile(
+ r"^([A-Za-z_][A-Za-z0-9_]*::[A-Za-z_][A-Za-z0-9_]*)\s*:\s*(OK|FAIL|ERROR)$", re.MULTILINE
+ )
+
+ for match in test_line_pattern.finditer(text_output):
+ test_name = match.group(1).strip()
+ status = match.group(2).strip()
+
+ if status == "OK":
+ results[test_name] = "PASSED"
+ elif status in ["FAIL", "ERROR"]:
+ results[test_name] = "FAILED"
+
+ # PRIORITY 2: If we found NO test results, check for error indicators
+ # Only return None if we're certain tests didn't run (compilation/linking errors)
+ if not results:
+ error_indicators = [
+ "error:", # C++ compilation errors
+ "undefined reference to", # Linking errors
+ "fatal error:", # Fatal compilation errors
+ "ld returned", # Linker errors
+ "cannot find -l", # Library linking errors
+ ]
+ has_errors = any(indicator in text_output for indicator in error_indicators)
+ # Return None ONLY if: no results found AND errors present
+ # Return empty dict if: no results found AND no errors (rare but valid - no tests)
+ return None if has_errors else results
+
+ return results
+
+
+def parse_minitest_text(text_output: str, test_metadata_path: str = None) -> Dict[str, str]:
+ """
+ Parse mini.nvim (MiniTest) test framework output.
+
+ MiniTest is used by Neovim plugins for testing.
+ Example output:
+ Total number of cases: 5
+ tests/test_treesitter.lua: ooooo
+
+ Fails (0) and Notes (0)
+
+ Or with failures:
+ FAIL in tests/test_treesitter.lua | wrap_cursor | normal: error message
+ FAIL in tests/test_treesitter.lua | enumerate: error message
+
+ Fails (2) and Notes (0)
+
+ IMPORTANT: MiniTest only outputs individual test names when they FAIL.
+ When all tests pass, only summary is shown - no individual test names.
+
+ Solution: When all tests pass, read test_metadata.json to get expected test names
+ and return them as PASSED. This ensures real test names are used consistently.
+ """
+ results = {}
+
+ # Parse individual test results from FAIL/NOTE lines
+ # Format: FAIL in file.lua | group | test_name: error message
+ # Use [^|:]+ to stop at pipe OR colon (prevents capturing error message)
+ fail_pattern = re.compile(
+ r"^(?:\x1b\[\d+(?:;\d+)?m)?FAIL(?:\x1b\[0m)?\s+in\s+([^|]+)\s*\|\s*([^|:]+)(?:\s*\|\s*([^:]+))?:", re.MULTILINE
+ )
+
+ for match in fail_pattern.finditer(text_output):
+ file_path = match.group(1).strip()
+ group = match.group(2).strip()
+ test_name = match.group(3).strip() if match.group(3) else ""
+
+ # Create test ID: file | group | test_name or file | group
+ if test_name:
+ test_id = f"{file_path} | {group} | {test_name}"
+ else:
+ test_id = f"{file_path} | {group}"
+
+ results[test_id] = "FAILED"
+
+ return results
+
+
+def parse_telescope_text(text_output: str) -> Dict[str, str]:
+ """
+ Parse telescope test framework output.
+
+ Telescope outputs lines like:
+ ✓ test_name
+ ✗ test_name
+ - test_name (skipped)
+
+ Also handles PlenaryBusted output for Neovim plugins.
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+ We only return None if we're certain the framework didn't run.
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If telescope failed to run (not the same as tests failing)
+ """
+ results = {}
+
+ for line in text_output.split("\n"):
+ line = line.strip()
+
+ # Match passed tests: ✓ test_name or "Success: test_name"
+ if "✓" in line:
+ test_name = line.split("✓", 1)[1].strip()
+ if test_name: # Avoid empty test names
+ results[test_name] = "PASSED"
+ elif line.lower().startswith("success:"):
+ test_name = line.split(":", 1)[1].strip()
+ if test_name:
+ results[test_name] = "PASSED"
+
+ # Match failed tests: ✗ test_name or "Failed: test_name"
+ elif "✗" in line:
+ test_name = line.split("✗", 1)[1].strip()
+ if test_name:
+ results[test_name] = "FAILED"
+ elif line.lower().startswith("failed:"):
+ test_name = line.split(":", 1)[1].strip()
+ if test_name:
+ results[test_name] = "FAILED"
+
+ # Match skipped tests: - test_name or "Skipped: test_name"
+ elif line.startswith("- ") and "skip" in line.lower():
+ test_name = line[2:].strip()
+ # Remove "(skipped)" suffix if present
+ test_name = re.sub(r"\s*\(skipped\)\s*$", "", test_name, flags=re.IGNORECASE)
+ if test_name:
+ results[test_name] = "SKIPPED"
+ elif line.lower().startswith("skipped:"):
+ test_name = line.split(":", 1)[1].strip()
+ if test_name:
+ results[test_name] = "SKIPPED"
+
+ # If no results found, try parsing summary line
+ if not results:
+ # Look for summary like "5 passed, 2 failed, 1 skipped"
+ summary_pattern = r"(\d+)\s+passed.*?(\d+)\s+failed"
+ match = re.search(summary_pattern, text_output.lower())
+ if match:
+ passed = int(match.group(1))
+ failed = int(match.group(2))
+
+ # Generate generic test IDs
+ for i in range(passed):
+ results[f"test_{i + 1}"] = "PASSED"
+ for i in range(failed):
+ results[f"test_failed_{i + 1}"] = "FAILED"
+
+ # PRIORITY 2: If we found NO test results, check for error indicators
+ # Only return None if we're certain tests didn't run (Lua/Neovim errors)
+ if not results:
+ error_indicators = [
+ "Error:", # Generic Lua errors
+ "error loading module", # Lua module loading errors
+ "attempt to call", # Lua runtime errors
+ "bad argument", # Lua runtime errors
+ "stack traceback:", # Lua errors with traceback
+ ]
+ has_errors = any(indicator in text_output for indicator in error_indicators)
+ # Return None ONLY if: no results found AND errors present
+ # Return empty dict if: no results found AND no errors (rare but valid - no tests)
+ return None if has_errors else results
+
+ return results
+
+
+def parse_lust_text(text_output: str) -> Dict[str, str]:
+ """
+ Parse lust test framework output.
+
+ Lust outputs test results with dots (.) for pass, F for fail.
+ Example output:
+ ..F.
+ 4 tests, 1 failure
+ test/my_test.lua:15: Expected true but got false
+
+ We parse individual test results when available, or fall back to summary.
+ """
+ results = {}
+
+ # Try to parse individual test results from verbose output
+ # Pattern: " test_name ... ok" or " test_name ... FAILED"
+ test_pattern = re.compile(r"^\s*(.+?)\s+\.\.\.\s+(ok|FAILED|ERROR)", re.MULTILINE)
+ matches = test_pattern.findall(text_output)
+
+ if matches:
+ # Found individual test results
+ for test_name, status in matches:
+ test_name = test_name.strip()
+ if status == "ok":
+ results[test_name] = "PASSED"
+ else:
+ results[test_name] = "FAILED"
+ return results
+
+ # Try to extract test descriptions from failure messages
+ # Pattern: "test_file.lua:line_number: test description"
+ failure_pattern = re.compile(r"^([^\s:]+\.lua):(\d+):\s*(.+)$", re.MULTILINE)
+ failures = failure_pattern.findall(text_output)
+
+ if failures:
+ for filepath, _, description in failures:
+ test_id = f"{filepath}::{description.strip()}"
+ results[test_id] = "FAILED"
+
+ # Parse summary line to get total count: "X tests, Y failures"
+ summary_match = re.search(r"(\d+)\s+tests?,\s+(\d+)\s+failures?", text_output.lower())
+ if summary_match:
+ total_tests = int(summary_match.group(1))
+ failures = int(summary_match.group(2))
+
+ # If we haven't parsed individual tests yet, generate generic ones
+ if not results:
+ passed = total_tests - failures
+ for i in range(passed):
+ results[f"test_{i + 1}"] = "PASSED"
+ for i in range(failures):
+ results[f"test_failed_{i + 1}"] = "FAILED"
+ return results
+
+ # Fallback: if no detailed info, check for overall success/failure
+ if not results:
+ if "0 failures" in text_output.lower() or "0 errors" in text_output.lower():
+ results["test_suite"] = "PASSED"
+ else:
+ results["test_suite"] = "FAILED"
+
+ return results
+
+
+def parse_bespoke_libgeos(text_output: str) -> Dict[str, str]:
+ """Parse libgeos/GEOS test output format.
+
+ Format:
+ capi::GEOSBoundary: .
+ capi::GEOSBuffer: .....................
+ geos::operation::OverlayNGEmptyCoordDim: [1=F][2=F].[4=F][5=F][6=F]
+ geos::operation::buffer::BufferOp: ..........................[27=X]
+
+ Where:
+ - dots (.) = passing tests
+ - [N=F] = explicit failure markers
+ - [N=X] = exception markers (also failures)
+ - standalone F or X = failure/exception
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+ We only return None if we're certain the framework didn't run.
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If libgeos tests failed to run (not the same as tests failing)
+ """
+ results = {}
+
+ # PRIORITY 1: Parse individual test result lines
+ # Pattern: TestSuite::TestName: followed by dots, Fs, Xs, or [N=F]/[N=X] markers
+ # Example: capi::GEOSBoundary: .
+ # Example: geos::OverlayNGEmptyCoordDim: [1=F][2=F].[4=F]
+ # Example: geos::operation::buffer::BufferOp: ..........................[27=X]
+ test_line_pattern = re.compile(
+ r"^([a-zA-Z_][a-zA-Z0-9_:]*::[a-zA-Z_][a-zA-Z0-9_]*)\s*:\s*(.+?)(?:\n|$)", re.MULTILINE
+ )
+
+ for match in test_line_pattern.finditer(text_output):
+ test_id = match.group(1) # Full name like "capi::GEOSBoundary"
+ test_output_line = match.group(2) # Everything after the colon
+
+ # Check for failure markers:
+ # 1. [N=F] pattern (explicit failure notation)
+ # 2. [N=X] pattern (exception notation)
+ # 3. Standalone F or X characters
+ has_failure = bool(re.search(r"\[.*=[FX]\]|(? str:
+ """Normalize XCTest case identifiers from swift test console output."""
+ name = raw_name.strip()
+
+ # Typical format: -[Module.Class testMethod]
+ if name.startswith("-[") and name.endswith("]"):
+ inner = name[2:-1].strip()
+ if " " in inner:
+ class_name, method = inner.split(" ", 1)
+ return f"{class_name}::{method}"
+ return inner
+
+ # Alternate format: Module.Class.testMethod
+ if "." in name:
+ parts = name.split(".", 1)
+ return f"{parts[0]}::{parts[1]}"
+
+ return name
+
+
+def parse_swift_test_text(text_output: str) -> Dict[str, str]:
+ """Parse vanilla `swift test` console output (without --xunit-output)."""
+ results = {}
+ test_case_pattern = re.compile(r"Test Case '([^']+)' (passed|failed|skipped)", re.IGNORECASE)
+
+ for match in test_case_pattern.finditer(text_output):
+ raw_name, status = match.groups()
+ test_id = _normalize_swift_test_name(raw_name)
+ results[test_id] = status.upper()
+
+ if results:
+ return results
+
+ # Fallback: parse summary line to infer aggregate results if per-test lines missing
+ summary_match = re.search(
+ r"Executed\s+(\d+)\s+tests?,\s+with\s+(\d+)\s+failures?",
+ text_output,
+ re.IGNORECASE,
+ )
+ if summary_match:
+ total_tests = int(summary_match.group(1))
+ failures = int(summary_match.group(2))
+ passes = max(total_tests - failures, 0)
+
+ for i in range(passes):
+ results[f"swift_test_pass_{i + 1}"] = "PASSED"
+ for i in range(failures):
+ results[f"swift_test_fail_{i + 1}"] = "FAILED"
+
+ return results
+
+
+def parse_xctest_output(output: str) -> Dict[str, str]:
+ """Parse XCTest results, preferring XML when available."""
+ xml_results = parse_junit_xml(output)
+ if xml_results:
+ return xml_results
+ return parse_swift_test_text(output)
+
+
+def normalize_test_id(test_id: str, framework: str = "") -> str:
+ """Normalize test IDs for stable matching across different formats.
+
+ This function performs several normalizations:
+
+ 1. Removes unstable runtime prefixes that change between runs:
+ - (N/M) - Test execution order (e.g., "(2/5) test_name")
+ - [N/M] - Alternative bracket format
+ - #N - Test number prefix (e.g., "#42 test_name")
+ - N. - Numbered list format (e.g., "1. test_name")
+
+ 2. Removes common file extensions (.py, .js, .ts, .go, etc.) from test paths
+ to allow matching between "test_file.py::test" and "test_file::test"
+
+ 3. Normalizes delimiters (`.`, `::`, `/`) to a canonical form (`::`)
+ when they appear between alphanumeric characters, allowing matching
+ between "testa.testb::testc" and "testa/testb.testc"
+
+ Examples:
+ "(2/5) test_name" -> "test_name"
+ "test_file.py::test_name" -> "test_file::test_name"
+ "testa.testb::testc" -> "testa::testb::testc"
+ "testa/testb.testc" -> "testa::testb::testc"
+ "tests/module.js::describe::it" -> "tests::module::describe::it"
+
+ Args:
+ test_id: Original test ID from parser
+ framework: Test framework name (for future framework-specific rules if needed)
+
+ Returns:
+ Normalized test ID
+ """
+ # Step 1: Remove unstable runtime prefixes
+
+ # Universal pattern: Remove (N/M) or [N/M] prefixes (test execution order)
+ # Matches: "(2/5) test", "[2/5] test", "(123/456) test", "( 1/75) test" (with internal space)
+ normalized = re.sub(r"^[\(\[]?\s*\d+/\d+[\)\]]?\s+", "", test_id)
+
+ # Universal pattern: Remove #N prefix (test numbering)
+ # Matches: "#42 test", "# 42 test"
+ normalized = re.sub(r"^#\s*\d+\s+", "", normalized)
+
+ # Universal pattern: Remove "N. " prefix (numbered list)
+ # Matches: "1. test", "42. test"
+ normalized = re.sub(r"^\d+\.\s+", "", normalized)
+
+ # Step 2: Remove common file extensions before delimiters
+ # This prevents .py from becoming ::py after delimiter normalization
+ # Match extensions like .py, .js, .ts, etc. that appear before :: / . or end of string
+ extensions_pattern = (
+ r"\.(py|pyw|js|mjs|cjs|ts|mts|cts|jsx|tsx|"
+ r"go|java|rb|rs|c|cpp|cc|cxx|h|hpp|hxx|"
+ r"swift|kt|kts|scala|php|cs|fs|"
+ r"ex|exs|erl|hrl|clj|cljs|cljc|"
+ r"lua|pl|pm|t|r|R|m|mm|"
+ r"f|f90|f95|for|vb|pas|pp|"
+ r"d|nim|zig|v|sv|vhd|vhdl|"
+ r"tcl|sh|bash|zsh|fish|ps1|psm1|psd1)"
+ r"(?=::|/|\.|$)"
+ )
+ normalized = re.sub(extensions_pattern, "", normalized, flags=re.IGNORECASE)
+
+ # Step 3: Normalize delimiters (., ::, /) to :: when between word characters
+ # This allows matching "testa.testb::testc" with "testa/testb.testc"
+ delimiter_pattern = r"(?<=\w)(::|\.|/)(?=\w)"
+ normalized = re.sub(delimiter_pattern, "::", normalized)
+
+ return normalized
+
+
+def parse_tap_text(text_output: str) -> Dict[str, str]:
+ """
+ Parse TAP (Test Anything Protocol) output.
+
+ TAP is used by tape, node-tap, and other JavaScript test frameworks.
+
+ Format:
+ TAP version 13
+ # Subtest: Test name
+ 1..N
+ ok 1 - assertion name
+ not ok 2 - assertion name
+ ok 1 - Test name # time=123ms
+ not ok 2 - Test name
+ 1..N
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+ We only return None if we're certain the tests didn't run.
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If TAP tests failed to run (not the same as tests failing)
+ """
+ results = {}
+
+ # PRIORITY 1: Parse top-level test results (not indented subtests)
+ # Format: "ok N - Test name" or "not ok N - Test name"
+ # Skip lines starting with whitespace (subtests)
+ tap_test_pattern = re.compile(
+ r"^(not )?ok\s+(\d+)\s*(?:-\s*)?(.+?)(?:\s*#\s*(skip|todo|time=.*))?$", re.MULTILINE | re.IGNORECASE
+ )
+
+ for match in tap_test_pattern.finditer(text_output):
+ is_failure = match.group(1) is not None # "not ok" prefix
+ test_num = match.group(2)
+ test_name = match.group(3).strip() if match.group(3) else f"test_{test_num}"
+ directive = match.group(4)
+
+ # Clean up test name (remove timing info like "# time=123ms")
+ test_name = re.sub(r"\s*#\s*time=[\d.]+m?s\s*$", "", test_name, flags=re.IGNORECASE)
+
+ test_id = test_name if test_name else f"test_{test_num}"
+
+ # Check for skip directive
+ if directive and directive.lower().startswith("skip"):
+ results[test_id] = "SKIPPED"
+ elif is_failure:
+ results[test_id] = "FAILED"
+ else:
+ results[test_id] = "PASSED"
+
+ # PRIORITY 2: If no results found, try parsing summary line
+ # Format: "# tests N", "# pass N", "# fail N"
+ if not results:
+ pass_match = re.search(r"#\s*pass\s+(\d+)", text_output, re.IGNORECASE)
+ fail_match = re.search(r"#\s*fail\s+(\d+)", text_output, re.IGNORECASE)
+
+ if pass_match or fail_match:
+ passed = int(pass_match.group(1)) if pass_match else 0
+ failed = int(fail_match.group(1)) if fail_match else 0
+
+ for i in range(passed):
+ results[f"tap_test_passed_{i + 1}"] = "PASSED"
+ for i in range(failed):
+ results[f"tap_test_failed_{i + 1}"] = "FAILED"
+
+ # PRIORITY 3: If no results, check for error indicators
+ if not results:
+ error_indicators = [
+ "npm ERR!", # npm errors
+ "Error: Cannot find module", # Module loading errors
+ "SyntaxError:", # JavaScript syntax errors
+ "TypeError:", # Type errors
+ ]
+ has_errors = any(indicator in text_output for indicator in error_indicators)
+ # Return None ONLY if: no results found AND errors present
+ return None if has_errors else results
+
+ return results
+
+
+def parse_hardhat_mocha_text(text_output: str) -> Dict[str, str]:
+ """
+ Parse Hardhat/Mocha console text output (non-JSON reporter).
+
+ Hardhat uses Mocha under the hood and outputs text like:
+ Contract: FeeSharingProxy:
+ withdrawFees
+ ✓ Shouldn't be able to use zero token address
+ ✓ Shouldn't be able to withdraw second time in period
+ 1) Should fail with specific error
+
+ 5 passing (1s)
+ 1 failing
+
+ IMPORTANT: We prioritize finding valid test results over detecting errors.
+
+ Returns:
+ Dict[str, str]: Test results mapping test IDs to status (PASSED/FAILED/SKIPPED)
+ None: If tests failed to run (not the same as tests failing)
+ """
+ results = {}
+
+ # Track current context (Contract/describe blocks)
+ current_context = []
+
+ # PRIORITY 1: Parse individual test results
+ for line in text_output.split("\n"):
+ stripped = line.strip()
+
+ # Track Contract: or describe blocks
+ contract_match = re.match(r"^Contract:\s*(.+?):\s*$", stripped)
+ if contract_match:
+ current_context = [contract_match.group(1)]
+ continue
+
+ # Track describe blocks (indented without checkmark/number)
+ if stripped and not stripped.startswith(("✓", "✗", "-")) and not re.match(r"^\d+\)", stripped):
+ # Check if this looks like a describe block (usually followed by test cases)
+ if ":" not in stripped and len(stripped) < 100:
+ # This might be a describe block, but we'll handle it dynamically
+ pass
+
+ # Match passed tests: ✓ test_name or ✔ test_name
+ pass_match = re.match(r"^[✓✔]\s+(.+?)(?:\s+\(\d+m?s\))?$", stripped)
+ if pass_match:
+ test_name = pass_match.group(1).strip()
+ test_id = f"{' > '.join(current_context)} > {test_name}" if current_context else test_name
+ results[test_id] = "PASSED"
+ continue
+
+ # Match failed tests: N) test_name or ✗ test_name
+ fail_match = re.match(r"^(?:\d+\)|[✗✘])\s*(.+?)$", stripped)
+ if fail_match:
+ test_name = fail_match.group(1).strip()
+ test_id = f"{' > '.join(current_context)} > {test_name}" if current_context else test_name
+ results[test_id] = "FAILED"
+ continue
+
+ # Match skipped tests: - test_name
+ skip_match = re.match(r"^-\s+(.+?)$", stripped)
+ if skip_match:
+ test_name = skip_match.group(1).strip()
+ test_id = f"{' > '.join(current_context)} > {test_name}" if current_context else test_name
+ results[test_id] = "SKIPPED"
+ continue
+
+ # PRIORITY 2: Parse summary if no individual results found
+ if not results:
+ # Look for "N passing" and "N failing"
+ pass_match = re.search(r"(\d+)\s+passing", text_output, re.IGNORECASE)
+ fail_match = re.search(r"(\d+)\s+failing", text_output, re.IGNORECASE)
+
+ if pass_match or fail_match:
+ passed = int(pass_match.group(1)) if pass_match else 0
+ failed = int(fail_match.group(1)) if fail_match else 0
+
+ for i in range(passed):
+ results[f"mocha_test_passed_{i + 1}"] = "PASSED"
+ for i in range(failed):
+ results[f"mocha_test_failed_{i + 1}"] = "FAILED"
+
+ # PRIORITY 3: If no results, check for error indicators
+ if not results:
+ error_indicators = [
+ "Error: Cannot find module",
+ "SyntaxError:",
+ "CompilerError:", # Solidity compilation errors
+ "Error: HH", # Hardhat errors
+ ]
+ has_errors = any(indicator in text_output for indicator in error_indicators)
+ return None if has_errors else results
+
+ return results
+
+
+def parse_pytest_text(text_output: str) -> Dict[str, str]:
+ """
+ Parse pytest plain text output (-v flag).
+
+ Pytest outputs lines like:
+ tests/test_foo.py::test_one PASSED
+ tests/test_foo.py::test_two FAILED
+ tests/test_foo.py::test_three SKIPPED
+
+ Or in short form:
+ tests/test_foo.py .F.s
+
+ Also handles summary lines like:
+ ===== 3 passed, 1 failed, 1 skipped in 0.5s =====
+ """
+ results = {}
+
+ # Pattern 1: Verbose output with test names
+ # e.g., "tests/test_foo.py::test_one PASSED"
+ verbose_pattern = re.compile(
+ r"^([\w./]+::\w+(?:::\w+)*)\s+(PASSED|FAILED|SKIPPED|ERROR|XFAIL|XPASS)", re.MULTILINE
+ )
+
+ for match in verbose_pattern.finditer(text_output):
+ test_id = match.group(1).strip()
+ status = match.group(2).upper()
+
+ if status in ("PASSED", "XPASS"):
+ results[test_id] = "PASSED"
+ elif status in ("FAILED", "ERROR", "XFAIL"):
+ results[test_id] = "FAILED"
+ elif status == "SKIPPED":
+ results[test_id] = "SKIPPED"
+
+ if results:
+ return results
+
+ # Pattern 2: Short form with dots (. = pass, F = fail, s = skip)
+ # e.g., "tests/test_foo.py .F.s"
+ short_pattern = re.compile(r"^([\w./]+\.py)\s+([.FsExX]+)", re.MULTILINE)
+
+ for match in short_pattern.finditer(text_output):
+ file_path = match.group(1)
+ outcomes = match.group(2)
+
+ for i, char in enumerate(outcomes):
+ test_id = f"{file_path}::test_{i + 1}"
+ if char == ".":
+ results[test_id] = "PASSED"
+ elif char.upper() == "F":
+ results[test_id] = "FAILED"
+ elif char.lower() == "s":
+ results[test_id] = "SKIPPED"
+
+ if results:
+ return results
+
+ # Pattern 3: Summary line fallback
+ # e.g., "===== 3 passed, 1 failed, 1 skipped in 0.5s ====="
+ summary_pattern = re.compile(r"(\d+)\s+passed(?:,\s*(\d+)\s+failed)?(?:,\s*(\d+)\s+(?:skipped|deselected))?")
+ match = summary_pattern.search(text_output)
+ if match:
+ passed = int(match.group(1) or 0)
+ failed = int(match.group(2) or 0)
+ skipped = int(match.group(3) or 0)
+
+ for i in range(passed):
+ results[f"pytest_test_passed_{i + 1}"] = "PASSED"
+ for i in range(failed):
+ results[f"pytest_test_failed_{i + 1}"] = "FAILED"
+ for i in range(skipped):
+ results[f"pytest_test_skipped_{i + 1}"] = "SKIPPED"
+
+ return results
+
+
+def parse_test_output(output: str, framework: str) -> Dict[str, str]:
+ """
+ Parse test output to extract individual test results.
+
+ Returns: {'test_id': 'PASSED'|'FAILED'|'SKIPPED'}
+ """
+ # Direct framework → parser mapping
+ parsers = {
+ "pytest": parse_junit_xml,
+ "unittest": parse_junit_xml,
+ "junit": parse_junit_xml,
+ "maven": parse_maven_text_output,
+ "gtest": parse_gtest_json,
+ "cargo-nextest": parse_cargo_nextest,
+ "go": parse_go_json,
+ "jest": parse_jest_vitest_json,
+ "vitest": parse_jest_vitest_json,
+ "mocha": parse_mocha_json,
+ "bun": parse_bun_text,
+ "ctest": parse_junit_xml,
+ "cppunit": parse_cppunit_text,
+ "bespoke_libgeos": parse_bespoke_libgeos,
+ # XCTest using hybrid approach
+ "xctest": parse_xctest_output,
+ "testing": parse_xctest_output, # New Swift Testing framework (Swift 6+)
+ # Lua frameworks
+ "busted": parse_junit_xml, # Uses JUnit XML output
+ "luaunit": parse_junit_xml, # Uses JUnit XML output
+ "telescope": parse_telescope_text,
+ "lust": parse_lust_text,
+ "minitest": parse_minitest_text, # Neovim mini.nvim test framework
+ # TAP (Test Anything Protocol) - used by tape, node-tap
+ "tap": parse_tap_text,
+ "tape": parse_tap_text,
+ # Hardhat (Solidity) - uses Mocha console output
+ "hardhat": parse_hardhat_mocha_text,
+ }
+
+ parser = parsers.get(framework)
+ if parser:
+ result = parser(output)
+ # Fallback for common frameworks if their primary parser returns None/empty
+ if not result:
+ if framework in ["junit", "maven"]:
+ result = parse_maven_text_output(output)
+ elif framework == "pytest":
+ # Pytest often outputs plain text, not JUnit XML
+ result = parse_pytest_text(output)
+ elif framework == "mocha":
+ # Mocha might output text instead of JSON (console reporter)
+ result = parse_hardhat_mocha_text(output)
+ return result or {}
+
+ # Try auto-detection for unknown frameworks
+ # Check for TAP output
+ if "TAP version" in output or re.search(r"^(?:not )?ok\s+\d+", output, re.MULTILINE):
+ return parse_tap_text(output) or {}
+
+ # Check for Mocha/Hardhat console output
+ if "Contract:" in output or re.search(r"^\s*[✓✔]\s+", output, re.MULTILINE):
+ return parse_hardhat_mocha_text(output) or {}
+
+ return {}
diff --git a/responses_api_agents/swe_env/parsing/utils.py b/responses_api_agents/swe_env/parsing/utils.py
new file mode 100644
index 0000000000..e352835284
--- /dev/null
+++ b/responses_api_agents/swe_env/parsing/utils.py
@@ -0,0 +1,194 @@
+# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""SWE-Bench-Ext test output parsing utilities.
+
+Provides the high-level grading entry point that parses raw test output,
+normalizes test IDs, fuzzy-matches the expected FAIL_TO_PASS / PASS_TO_PASS
+tests, and reports whether the task was resolved. Example usage::
+
+ from responses_api_agents.swe_env.parsing import parse_and_check_tests
+
+ result = parse_and_check_tests(
+ test_output=log_text,
+ test_framework="pytest",
+ fail_to_pass=["test_a", "test_b"],
+ pass_to_pass=["test_c"],
+ instance_id="my-task-123",
+ )
+ # result["resolved"] -> bool
+"""
+
+from __future__ import annotations
+
+from typing import Any, Dict, List, Optional
+
+from responses_api_agents.swe_env.parsing.parsing import (
+ normalize_test_id,
+ parse_test_output,
+)
+
+
+# Marker strings used to delimit structured output in the raw test log.
+_TEST_OUTPUT_START = "<<>>"
+_TEST_OUTPUT_END = "<<>>"
+_RESULT_FILE_START = "<<>>"
+_RESULT_FILE_END = "<<>>"
+
+
+def _extract_between_markers(text: str, start: str, end: str) -> Optional[str]:
+ """Extract the substring between two marker strings.
+
+ Args:
+ text: Text to search within.
+ start: Opening marker; the result begins after it.
+ end: Closing marker; the result ends before it.
+
+ Returns:
+ The stripped text between the markers, or None if either marker is
+ missing or they appear out of order.
+ """
+ s = text.find(start)
+ e = text.find(end)
+ if s != -1 and e != -1 and s < e:
+ return text[s + len(start) : e].strip()
+ return None
+
+
+def _match_test_with_fuzzy(
+ test_id: str,
+ parsed_results: Dict[str, str],
+ build_failed_packages: set,
+) -> str:
+ """Resolve the status of a single test ID against parsed results.
+
+ Tries, in order: a direct lookup, a check for membership in a package that
+ failed to build, a substring match, and a match on the final ``::``
+ component.
+
+ Args:
+ test_id: Normalized test identifier to look up.
+ parsed_results: Mapping of parsed test ID to status string.
+ build_failed_packages: Set of package names whose build failed; any
+ test ID prefixed by one of these is treated as failed.
+
+ Returns:
+ The matched status string, ``"FAILED"`` if the test's package failed to
+ build, or ``"NOT_FOUND"`` if no match is found.
+ """
+ # Direct match
+ if test_id in parsed_results:
+ return parsed_results[test_id]
+
+ # Check if this test belongs to a package that failed to build
+ for pkg in build_failed_packages:
+ if test_id.startswith(pkg):
+ return "FAILED"
+
+ # Substring match (normalized IDs may differ in prefix)
+ for parsed_id, status in parsed_results.items():
+ if test_id in parsed_id or parsed_id in test_id:
+ return status
+
+ # Try matching by last component (after last ::)
+ if "::" in test_id:
+ suffix = test_id.rsplit("::", 1)[-1]
+ for parsed_id, status in parsed_results.items():
+ if "::" in parsed_id and parsed_id.rsplit("::", 1)[-1] == suffix:
+ return status
+
+ return "NOT_FOUND"
+
+
+def parse_and_check_tests(
+ test_output: str,
+ test_framework: str,
+ fail_to_pass: List[str],
+ pass_to_pass: List[str],
+ instance_id: str = "",
+) -> Dict[str, Any]:
+ """Parse test output and check FAIL_TO_PASS / PASS_TO_PASS resolution.
+
+ The pipeline extracts structured output from the result-file markers (if
+ present), parses it with the framework dispatcher, normalizes both parsed
+ and expected test IDs, fuzzy-matches each expected test, and computes
+ ``resolved`` as all FAIL_TO_PASS passing and all PASS_TO_PASS passing.
+
+ Args:
+ test_output: Raw test log to parse.
+ test_framework: Name of the test framework (e.g. ``"pytest"``) used to
+ select the parser and normalize IDs.
+ fail_to_pass: Test IDs expected to transition from failing to passing.
+ pass_to_pass: Test IDs expected to remain passing.
+ instance_id: Optional task identifier, accepted for caller convenience.
+
+ Returns:
+ A report dict containing the overall ``resolved`` flag, per-test
+ FAIL_TO_PASS and PASS_TO_PASS results, pass/total counts for each
+ group, the number of parsed tests, and the framework name.
+ """
+ # Try to extract result file content from the markers.
+ result_file_content = _extract_between_markers(test_output, _RESULT_FILE_START, _RESULT_FILE_END)
+
+ if result_file_content:
+ parsed = parse_test_output(result_file_content, test_framework)
+ if not parsed:
+ parsed = parse_test_output(test_output, test_framework)
+ else:
+ parsed = parse_test_output(test_output, test_framework)
+
+ if parsed is None:
+ parsed = {}
+
+ # Normalize parsed test IDs
+ parsed = {normalize_test_id(tid, test_framework): status for tid, status in parsed.items()}
+
+ # Normalize expected test IDs
+ norm_f2p = [normalize_test_id(tid, test_framework) for tid in fail_to_pass]
+ norm_p2p = [normalize_test_id(tid, test_framework) for tid in pass_to_pass]
+
+ # Handle synthetic build/compile tests
+ for tid in norm_f2p + norm_p2p:
+ if (tid.endswith("::build") or tid.endswith("::compile")) and tid not in parsed:
+ parsed[tid] = "PASSED"
+
+ # Identify packages that failed to build
+ build_failed_packages = {pkg for pkg, status in parsed.items() if status == "FAILED" and "::" not in pkg}
+
+ # Match FAIL_TO_PASS
+ f2p_results = {}
+ for tid in norm_f2p:
+ f2p_results[tid] = _match_test_with_fuzzy(tid, parsed, build_failed_packages)
+
+ # Match PASS_TO_PASS
+ p2p_results = {}
+ for tid in norm_p2p:
+ p2p_results[tid] = _match_test_with_fuzzy(tid, parsed, build_failed_packages)
+
+ all_f2p_passed = all(v == "PASSED" for v in f2p_results.values()) if f2p_results else False
+ all_p2p_passed = all(v == "PASSED" for v in p2p_results.values())
+ resolved = all_f2p_passed and all_p2p_passed
+
+ return {
+ "resolved": resolved,
+ "patch_exists": True,
+ "patch_successfully_applied": True,
+ "fail_to_pass_results": f2p_results,
+ "pass_to_pass_results": p2p_results,
+ "f2p_passed": sum(1 for v in f2p_results.values() if v == "PASSED"),
+ "f2p_total": len(f2p_results),
+ "p2p_passed": sum(1 for v in p2p_results.values() if v == "PASSED"),
+ "p2p_total": len(p2p_results),
+ "parsed_count": len(parsed),
+ "framework": test_framework,
+ }
diff --git a/responses_api_agents/swe_env/providers/__init__.py b/responses_api_agents/swe_env/providers/__init__.py
new file mode 100644
index 0000000000..9ce6c534f2
--- /dev/null
+++ b/responses_api_agents/swe_env/providers/__init__.py
@@ -0,0 +1,45 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""SWE-env sandbox providers.
+
+Importing this package registers the providers with ``nemo_gym.sandbox`` so a
+config like ``provider: {docker: {...}}`` resolves. ``docker`` runs sandboxes
+as local Docker containers; ``apptainer`` runs them as ``apptainer instance``
+processes from ``.sif`` images for on-prem clusters.
+"""
+
+from nemo_gym.sandbox import list_providers, register_provider
+from responses_api_agents.swe_env.providers.apptainer_provider import ApptainerSandboxProvider
+from responses_api_agents.swe_env.providers.docker_provider import DockerSandboxProvider
+
+
+def register_swe_env_providers() -> None:
+ """Register the swe_env sandbox providers, skipping any already registered.
+
+ Registers the ``docker`` and ``apptainer`` providers with
+ ``nemo_gym.sandbox``. Safe to call multiple times; providers already present
+ in the registry are left untouched.
+ """
+ existing = set(list_providers())
+ if "docker" not in existing:
+ register_provider("docker", DockerSandboxProvider)
+ if "apptainer" not in existing:
+ register_provider("apptainer", ApptainerSandboxProvider)
+
+
+register_swe_env_providers()
+
+
+__all__ = ["ApptainerSandboxProvider", "DockerSandboxProvider", "register_swe_env_providers"]
diff --git a/responses_api_agents/swe_env/providers/apptainer_provider.py b/responses_api_agents/swe_env/providers/apptainer_provider.py
new file mode 100644
index 0000000000..1555f64cd7
--- /dev/null
+++ b/responses_api_agents/swe_env/providers/apptainer_provider.py
@@ -0,0 +1,346 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Apptainer-backed ``SandboxProvider`` for ``.sif`` images.
+
+Implements the ``nemo_gym.sandbox`` provider Protocol using a long-lived
+``apptainer instance`` so repository edits persist across exec calls, and a
+bind-mounted host scratch directory for transferring files in and out of the
+sandbox.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import glob
+import os
+import posixpath
+import shlex
+import shutil
+import tempfile
+import uuid
+from pathlib import Path
+from typing import Any
+
+from nemo_gym.sandbox import (
+ SandboxCreateError,
+ SandboxExecResult,
+ SandboxHandle,
+ SandboxSpec,
+ SandboxStatus,
+)
+
+
+_IO_MOUNT = "/sandbox_io"
+
+
+class ApptainerSandboxProvider:
+ """Run sandboxes as ``apptainer instance`` processes from ``.sif`` images."""
+
+ name = "apptainer"
+
+ def __init__(
+ self,
+ *,
+ apptainer_bin: str = "apptainer",
+ image_root: str | None = None,
+ scratch_root: str | None = None,
+ instance_args: list[str] | None = None,
+ exec_args: list[str] | None = None,
+ **_: Any,
+ ) -> None:
+ """Configure the Apptainer sandbox provider.
+
+ Args:
+ apptainer_bin: Name or path of the ``apptainer`` executable.
+ image_root: Optional root directory searched for ``.sif`` images when
+ a spec does not point at an existing file.
+ scratch_root: Optional parent directory for the per-sandbox scratch
+ directory used as the bind-mounted I/O area.
+ instance_args: Flags passed to ``apptainer instance start``. Defaults
+ to ``--writable-tmpfs --cleanenv --pid --no-mount
+ home,tmp,bind-paths`` so the nested harness gets its own PID
+ namespace and the host ``$HOME``/``tmp`` plus the image's
+ declared bind points stay out of the sandbox, letting the
+ prebuilt venv resolve against the explicit ``--bind`` mounts.
+ exec_args: Extra flags passed to every ``apptainer exec``.
+ **_: Additional keyword arguments are accepted and ignored.
+ """
+ self._bin = apptainer_bin
+ self._image_root = image_root
+ self._scratch_root = scratch_root
+ self._instance_args = list(
+ instance_args
+ if instance_args is not None
+ else ["--writable-tmpfs", "--cleanenv", "--pid", "--no-mount", "home,tmp,bind-paths"]
+ )
+ self._exec_args = list(exec_args or [])
+
+ async def _run(self, *args: str, timeout_s: int | float | None = None) -> tuple[int, str, str]:
+ """Run the ``apptainer`` CLI with the given arguments and capture output.
+
+ Args:
+ *args: Arguments passed to the ``apptainer`` executable.
+ timeout_s: Optional timeout in seconds; the process is killed and the
+ timeout error re-raised if it is exceeded.
+
+ Returns:
+ A tuple of ``(return_code, stdout, stderr)`` with output decoded as
+ text using ``errors="replace"``.
+ """
+ proc = await asyncio.create_subprocess_exec(
+ self._bin, *args, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
+ )
+ try:
+ out, err = await asyncio.wait_for(proc.communicate(), timeout=timeout_s)
+ except (asyncio.TimeoutError, TimeoutError):
+ proc.kill()
+ await proc.wait()
+ raise
+ rc = proc.returncode if proc.returncode is not None else -1
+ return rc, out.decode(errors="replace"), err.decode(errors="replace")
+
+ def _resolve_sif(self, spec: SandboxSpec) -> str:
+ """Resolve a ``.sif`` image path from provider options or by globbing.
+
+ Prefers ``provider_options['sif_path']`` (or ``spec.image``) when it
+ names an existing file; otherwise globs under the configured image
+ roots, restricting matches to ``*.sif`` and also trying a lowercased
+ search term.
+
+ Args:
+ spec: Sandbox spec carrying the image name and provider options
+ (``sif_path``, ``image_glob``, ``image_root``).
+
+ Returns:
+ The resolved ``.sif`` path; when several candidates match, the last
+ in sorted order is chosen.
+
+ Raises:
+ SandboxCreateError: If no matching ``.sif`` file is found.
+ """
+ sif = spec.provider_options.get("sif_path") or spec.image
+ if sif and os.path.isfile(sif):
+ return sif
+ # Glob under the image roots. Restrict the fuzzy term to ``*.sif`` so an
+ # unrelated host file can never be picked up as a container, and add a
+ # lowercased search term to match case-folded candidates.
+ pattern = spec.provider_options.get("image_glob")
+ roots = [r for r in (self._image_root, spec.provider_options.get("image_root")) if r]
+ candidates: list[str] = []
+ for root in roots:
+ if pattern:
+ candidates += glob.glob(os.path.join(root, pattern))
+ elif sif:
+ terms = [sif] if sif == sif.lower() else [sif, sif.lower()]
+ for term in terms:
+ candidates += glob.glob(os.path.join(root, f"*{term}*.sif"))
+ if not candidates:
+ raise SandboxCreateError(f"No .sif found for image={spec.image!r} (roots={roots}, glob={pattern!r})")
+ return sorted(candidates)[-1]
+
+ @staticmethod
+ def _mount_binds(spec: SandboxSpec) -> list[str]:
+ """Translate ``provider_options['mounts']`` into apptainer ``--bind`` args.
+
+ Each mount is ``{"src": host_path, "dst": container_path[, "ro": bool]}``.
+ A read-only flag appends ``:ro`` to the bind.
+
+ Args:
+ spec: Sandbox spec whose ``provider_options['mounts']`` lists the
+ bind mounts to apply.
+
+ Returns:
+ A flat list of ``--bind`` arguments suitable for the apptainer CLI.
+ """
+ binds: list[str] = []
+ for mount in spec.provider_options.get("mounts", []) or []:
+ src = mount.get("src")
+ dst = mount.get("dst")
+ if not src or not dst:
+ continue
+ # Skip a self-bind whose source does not exist on the host: when no
+ # real host dataset path is provided the caller falls back to
+ # ``src == dst ==`` the in-container dataset path, which is not a host
+ # path. Binding a missing host src would make apptainer create an
+ # empty dir there and shadow the real dataset. Real host binds
+ # (src != dst, or an existing self-bind) are unaffected.
+ if src == dst and not os.path.exists(src):
+ continue
+ bind = f"{src}:{dst}"
+ if mount.get("ro"):
+ bind += ":ro"
+ binds += ["--bind", bind]
+ return binds
+
+ async def create(self, spec: SandboxSpec) -> SandboxHandle:
+ """Start an apptainer instance and return a handle to it.
+
+ Resolves the ``.sif`` image, creates a host scratch directory bound at
+ the in-container I/O mount, applies the configured instance args, extra
+ binds, and environment, and launches a named instance.
+
+ Args:
+ spec: Sandbox spec describing the image, env, workdir, provider
+ options, and readiness timeout.
+
+ Returns:
+ A ``SandboxHandle`` whose ``sandbox_id`` is the instance name; its
+ ``raw`` records the resolved sif, scratch dir, and workdir.
+
+ Raises:
+ SandboxCreateError: If starting the instance times out or fails; the
+ scratch directory is removed in that case.
+ """
+ sif = self._resolve_sif(spec)
+ scratch = tempfile.mkdtemp(prefix="swe-apptainer-io-", dir=self._scratch_root)
+ instance_name = f"swe-{(spec.metadata.get('instance_id') or 'task')[:24]}-{uuid.uuid4().hex[:8]}"
+ args = ["instance", "start", *self._instance_args, "--bind", f"{scratch}:{_IO_MOUNT}"]
+ args += self._mount_binds(spec)
+ for key, value in (spec.env or {}).items():
+ args += ["--env", f"{key}={value}"]
+ args += spec.provider_options.get("instance_args", [])
+ args += [sif, instance_name]
+ try:
+ rc, out, err = await self._run(*args, timeout_s=spec.ready_timeout_s or 600)
+ except (asyncio.TimeoutError, TimeoutError) as exc:
+ shutil.rmtree(scratch, ignore_errors=True)
+ raise SandboxCreateError(f"apptainer instance start timed out for {sif!r}") from exc
+ if rc != 0:
+ shutil.rmtree(scratch, ignore_errors=True)
+ raise SandboxCreateError(f"apptainer instance start failed (rc={rc}): {err.strip() or out.strip()}")
+ return SandboxHandle(
+ sandbox_id=instance_name,
+ provider_name=self.name,
+ raw={"sif": sif, "scratch": scratch, "workdir": spec.workdir},
+ )
+
+ async def exec(
+ self,
+ handle: SandboxHandle,
+ command: str,
+ *,
+ cwd: str | None = None,
+ env: dict[str, str] | None = None,
+ timeout_s: int | float | None = None,
+ user: str | int | None = None,
+ ) -> SandboxExecResult:
+ """Run a shell command inside the apptainer instance.
+
+ Args:
+ handle: Handle identifying the target instance.
+ command: Shell command executed via ``bash -c``.
+ cwd: Working directory for the command; falls back to the workdir
+ recorded at create time.
+ env: Extra environment variables for the command.
+ timeout_s: Optional timeout in seconds; on expiry a result with
+ return code 124 and ``error_type="timeout"`` is returned.
+ user: Accepted for interface compatibility; not applied by this
+ provider.
+
+ Returns:
+ A ``SandboxExecResult`` with stdout, stderr, and the return code, or
+ a timeout result if the command exceeds ``timeout_s``.
+ """
+ args = ["exec", *self._exec_args]
+ workdir = cwd or handle.raw.get("workdir")
+ if workdir:
+ args += ["--pwd", workdir]
+ for key, value in (env or {}).items():
+ args += ["--env", f"{key}={value}"]
+ args += [f"instance://{handle.sandbox_id}", "bash", "-c", command]
+ try:
+ rc, out, err = await self._run(*args, timeout_s=timeout_s)
+ except (asyncio.TimeoutError, TimeoutError):
+ return SandboxExecResult(
+ stdout=None, stderr=f"command timed out after {timeout_s}s", return_code=124, error_type="timeout"
+ )
+ return SandboxExecResult(stdout=out, stderr=err, return_code=rc)
+
+ async def upload_file(self, handle: SandboxHandle, source_path: Path, target_path: str) -> None:
+ """Copy a host file into the instance via the scratch I/O mount.
+
+ The file is staged into the host scratch directory, then copied to the
+ target path inside the instance, creating parent dirs as needed.
+
+ Args:
+ handle: Handle identifying the target instance.
+ source_path: Path to the file on the host.
+ target_path: Destination path inside the instance.
+
+ Raises:
+ RuntimeError: If the in-container copy fails.
+ """
+ scratch = handle.raw["scratch"]
+ base = posixpath.basename(target_path)
+ shutil.copy(str(source_path), os.path.join(scratch, base))
+ parent = posixpath.dirname(target_path)
+ mkdir = f"mkdir -p {shlex.quote(parent)} && " if parent else ""
+ result = await self.exec(handle, f"{mkdir}cp {_IO_MOUNT}/{shlex.quote(base)} {shlex.quote(target_path)}")
+ if result.return_code != 0:
+ raise RuntimeError(f"apptainer upload copy failed: {result.stderr}")
+
+ async def download_file(self, handle: SandboxHandle, source_path: str, target_path: Path) -> None:
+ """Copy a file out of the instance to the host via the scratch I/O mount.
+
+ The file is copied inside the instance to the scratch I/O mount, then
+ copied from the host scratch directory to the target path.
+
+ Args:
+ handle: Handle identifying the source instance.
+ source_path: Path to the file inside the instance.
+ target_path: Destination path on the host; parent dirs are created.
+
+ Raises:
+ RuntimeError: If the in-container copy fails.
+ """
+ scratch = handle.raw["scratch"]
+ base = posixpath.basename(source_path)
+ result = await self.exec(handle, f"cp {shlex.quote(source_path)} {_IO_MOUNT}/{shlex.quote(base)}")
+ if result.return_code != 0:
+ raise RuntimeError(f"apptainer download copy failed: {result.stderr}")
+ target = Path(target_path)
+ target.parent.mkdir(parents=True, exist_ok=True)
+ shutil.copy(os.path.join(scratch, base), str(target))
+
+ async def status(self, handle: SandboxHandle) -> SandboxStatus:
+ """Report whether the instance is running.
+
+ Args:
+ handle: Handle identifying the instance to query.
+
+ Returns:
+ ``RUNNING`` if the instance name appears in ``apptainer instance
+ list`` output, ``STOPPED`` if not, or ``UNKNOWN`` if the command
+ fails.
+ """
+ rc, out, _ = await self._run("instance", "list", handle.sandbox_id)
+ if rc != 0:
+ return SandboxStatus.UNKNOWN
+ return SandboxStatus.RUNNING if handle.sandbox_id in out else SandboxStatus.STOPPED
+
+ async def close(self, handle: SandboxHandle) -> None:
+ """Stop the instance and remove its scratch directory.
+
+ Args:
+ handle: Handle identifying the instance to stop.
+ """
+ try:
+ await self._run("instance", "stop", handle.sandbox_id)
+ finally:
+ shutil.rmtree(handle.raw.get("scratch", ""), ignore_errors=True)
+
+ async def aclose(self) -> None:
+ """Release provider-level resources; this provider holds none."""
+ return None
diff --git a/responses_api_agents/swe_env/providers/docker_provider.py b/responses_api_agents/swe_env/providers/docker_provider.py
new file mode 100644
index 0000000000..a26a4c56b7
--- /dev/null
+++ b/responses_api_agents/swe_env/providers/docker_provider.py
@@ -0,0 +1,285 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Local Docker-backed ``SandboxProvider`` implementation.
+
+Implements the ``nemo_gym.sandbox`` provider Protocol via the ``docker`` CLI so
+SWE environments can be provisioned and graded on any machine with Docker
+installed, making end-to-end SWE-bench verification runnable on a single
+workstation.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import posixpath
+import shlex
+from collections.abc import Mapping
+from pathlib import Path
+from typing import Any
+
+from nemo_gym.sandbox import (
+ SandboxCreateError,
+ SandboxExecResult,
+ SandboxHandle,
+ SandboxResources,
+ SandboxSpec,
+ SandboxStatus,
+)
+
+
+class DockerSandboxProvider:
+ """Run sandboxes as long-lived Docker containers via the ``docker`` CLI."""
+
+ name = "docker"
+
+ def __init__(
+ self,
+ *,
+ docker_bin: str = "docker",
+ default_user: str | int | None = None,
+ network: str | None = None,
+ run_args: list[str] | None = None,
+ keep_alive_command: str = "sleep infinity",
+ **_: Any,
+ ) -> None:
+ """Configure the Docker sandbox provider.
+
+ Args:
+ docker_bin: Name or path of the ``docker`` executable to invoke.
+ default_user: Default user (name or UID) to run ``exec`` commands as
+ when no per-call user is given; None leaves the image default.
+ network: Docker network to attach containers to; None uses the
+ Docker default.
+ run_args: Extra arguments appended to every ``docker run``
+ invocation.
+ keep_alive_command: Command run as the container's entrypoint to keep
+ it alive for subsequent ``exec`` calls.
+ **_: Additional keyword arguments are accepted and ignored.
+ """
+ self._bin = docker_bin
+ self._default_user = default_user
+ self._network = network
+ self._run_args = list(run_args or [])
+ self._keep_alive = keep_alive_command
+
+ async def _run(self, *args: str, timeout_s: int | float | None = None) -> tuple[int, str, str]:
+ """Run the ``docker`` CLI with the given arguments and capture output.
+
+ Args:
+ *args: Arguments passed to the ``docker`` executable.
+ timeout_s: Optional timeout in seconds; the process is killed and the
+ timeout error re-raised if it is exceeded.
+
+ Returns:
+ A tuple of ``(return_code, stdout, stderr)`` with output decoded as
+ text using ``errors="replace"``.
+ """
+ proc = await asyncio.create_subprocess_exec(
+ self._bin,
+ *args,
+ stdout=asyncio.subprocess.PIPE,
+ stderr=asyncio.subprocess.PIPE,
+ )
+ try:
+ out, err = await asyncio.wait_for(proc.communicate(), timeout=timeout_s)
+ except (asyncio.TimeoutError, TimeoutError):
+ proc.kill()
+ await proc.wait()
+ raise
+ return (
+ proc.returncode if proc.returncode is not None else -1,
+ out.decode(errors="replace"),
+ err.decode(errors="replace"),
+ )
+
+ @staticmethod
+ def _resources(spec: SandboxSpec) -> SandboxResources:
+ """Coerce a spec's resource request into a ``SandboxResources``.
+
+ Args:
+ spec: Sandbox spec whose ``resources`` field is a
+ ``SandboxResources`` or a mapping.
+
+ Returns:
+ The spec's ``SandboxResources`` if already one, otherwise a
+ ``SandboxResources`` built from the mapping (or empty defaults).
+ """
+ if isinstance(spec.resources, SandboxResources):
+ return spec.resources
+ return SandboxResources.from_mapping(spec.resources if isinstance(spec.resources, Mapping) else {})
+
+ async def create(self, spec: SandboxSpec) -> SandboxHandle:
+ """Start a detached container and return a handle to it.
+
+ Applies resource limits, network, working directory, environment, and
+ extra run args from the spec, then launches the image running the
+ keep-alive command so the container persists for later ``exec`` calls.
+
+ Args:
+ spec: Sandbox spec describing the image, resources, workdir, env, and
+ readiness timeout.
+
+ Returns:
+ A ``SandboxHandle`` whose ``sandbox_id`` is the container id.
+
+ Raises:
+ SandboxCreateError: If no image is given, ``docker run`` times out or
+ fails, or no container id is returned.
+ """
+ if not spec.image:
+ raise SandboxCreateError("DockerSandboxProvider requires spec.image")
+ args = ["run", "-d", "--init"]
+ if self._network:
+ args += ["--network", self._network]
+ res = self._resources(spec)
+ if res.memory_mib:
+ args.append(f"--memory={int(res.memory_mib)}m")
+ if res.cpu:
+ args.append(f"--cpus={res.cpu}")
+ if res.gpu:
+ args.append("--gpus=all")
+ if spec.workdir:
+ args += ["-w", spec.workdir]
+ for key, value in (spec.env or {}).items():
+ args += ["-e", f"{key}={value}"]
+ args += self._run_args
+ args += [spec.image, "bash", "-c", self._keep_alive]
+ try:
+ rc, out, err = await self._run(*args, timeout_s=spec.ready_timeout_s or 600)
+ except (asyncio.TimeoutError, TimeoutError) as exc:
+ raise SandboxCreateError(f"docker run timed out for image {spec.image!r}") from exc
+ if rc != 0:
+ raise SandboxCreateError(f"docker run failed (rc={rc}) for {spec.image!r}: {err.strip() or out.strip()}")
+ container_id = out.strip().splitlines()[-1].strip()
+ if not container_id:
+ raise SandboxCreateError("docker run did not return a container id")
+ return SandboxHandle(
+ sandbox_id=container_id,
+ provider_name=self.name,
+ raw={"image": spec.image, "workdir": spec.workdir},
+ )
+
+ async def exec(
+ self,
+ handle: SandboxHandle,
+ command: str,
+ *,
+ cwd: str | None = None,
+ env: dict[str, str] | None = None,
+ timeout_s: int | float | None = None,
+ user: str | int | None = None,
+ ) -> SandboxExecResult:
+ """Run a shell command inside the container.
+
+ Args:
+ handle: Handle identifying the target container.
+ command: Shell command executed via ``bash -c``.
+ cwd: Working directory for the command; falls back to the workdir
+ recorded at create time.
+ env: Extra environment variables for the command.
+ timeout_s: Optional timeout in seconds; on expiry a result with
+ return code 124 and ``error_type="timeout"`` is returned.
+ user: User (name or UID) to run as; falls back to the provider's
+ default user.
+
+ Returns:
+ A ``SandboxExecResult`` with stdout, stderr, return code, and an
+ ``error_type`` of ``"sandbox"`` for docker-level failures (125/126/
+ 127 with no stdout), ``"timeout"`` on timeout, or None otherwise.
+ """
+ args = ["exec"]
+ workdir = cwd or handle.raw.get("workdir")
+ if workdir:
+ args += ["-w", workdir]
+ eff_user = user if user is not None else self._default_user
+ if eff_user is not None:
+ args += ["-u", str(eff_user)]
+ for key, value in (env or {}).items():
+ args += ["-e", f"{key}={value}"]
+ args += [handle.sandbox_id, "bash", "-c", command]
+ try:
+ rc, out, err = await self._run(*args, timeout_s=timeout_s)
+ except (asyncio.TimeoutError, TimeoutError):
+ return SandboxExecResult(
+ stdout=None,
+ stderr=f"command timed out after {timeout_s}s",
+ return_code=124,
+ error_type="timeout",
+ )
+ # docker exec returns 125/126/127 for docker-level failures (container gone, not executable).
+ error_type = "sandbox" if rc in (125, 126, 127) and not out else None
+ return SandboxExecResult(stdout=out, stderr=err, return_code=rc, error_type=error_type)
+
+ async def upload_file(self, handle: SandboxHandle, source_path: Path, target_path: str) -> None:
+ """Copy a host file into the container, creating parent dirs as needed.
+
+ Args:
+ handle: Handle identifying the target container.
+ source_path: Path to the file on the host.
+ target_path: Destination path inside the container.
+
+ Raises:
+ RuntimeError: If the ``docker cp`` upload fails.
+ """
+ parent = posixpath.dirname(target_path)
+ if parent:
+ await self.exec(handle, f"mkdir -p {shlex.quote(parent)}")
+ rc, out, err = await self._run("cp", str(source_path), f"{handle.sandbox_id}:{target_path}")
+ if rc != 0:
+ raise RuntimeError(f"docker cp upload failed: {err.strip() or out.strip()}")
+
+ async def download_file(self, handle: SandboxHandle, source_path: str, target_path: Path) -> None:
+ """Copy a file out of the container to the host.
+
+ Args:
+ handle: Handle identifying the source container.
+ source_path: Path to the file inside the container.
+ target_path: Destination path on the host; parent dirs are created.
+
+ Raises:
+ RuntimeError: If the ``docker cp`` download fails.
+ """
+ target = Path(target_path)
+ target.parent.mkdir(parents=True, exist_ok=True)
+ rc, out, err = await self._run("cp", f"{handle.sandbox_id}:{source_path}", str(target))
+ if rc != 0:
+ raise RuntimeError(f"docker cp download failed: {err.strip() or out.strip()}")
+
+ async def status(self, handle: SandboxHandle) -> SandboxStatus:
+ """Report whether the container is running.
+
+ Args:
+ handle: Handle identifying the container to inspect.
+
+ Returns:
+ ``RUNNING`` or ``STOPPED`` based on the container's running state,
+ or ``UNKNOWN`` if the inspect command fails.
+ """
+ rc, out, _ = await self._run("inspect", "-f", "{{.State.Running}}", handle.sandbox_id)
+ if rc != 0:
+ return SandboxStatus.UNKNOWN
+ return SandboxStatus.RUNNING if out.strip() == "true" else SandboxStatus.STOPPED
+
+ async def close(self, handle: SandboxHandle) -> None:
+ """Force-remove the container.
+
+ Args:
+ handle: Handle identifying the container to remove.
+ """
+ await self._run("rm", "-f", handle.sandbox_id)
+
+ async def aclose(self) -> None:
+ """Release provider-level resources; this provider holds none."""
+ return None
diff --git a/responses_api_agents/swe_env/registry.py b/responses_api_agents/swe_env/registry.py
new file mode 100644
index 0000000000..97e41cf64b
--- /dev/null
+++ b/responses_api_agents/swe_env/registry.py
@@ -0,0 +1,70 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Name-to-harness registry for dispatching tasks to their SWE harness."""
+
+from __future__ import annotations
+
+from responses_api_agents.swe_env.harness import SweTaskHarness
+
+
+_HARNESSES: dict[str, SweTaskHarness] = {}
+
+
+def register_harness(harness: SweTaskHarness, *, override: bool = False) -> None:
+ """Register a harness under its ``name``.
+
+ Args:
+ harness (SweTaskHarness): The harness to register. Its ``name`` must be
+ non-empty.
+ override (bool): If ``True``, replace an existing harness with the same
+ name instead of raising.
+
+ Raises:
+ ValueError: If the harness name is empty, or a harness with the same name
+ is already registered and ``override`` is ``False``.
+ """
+ if not harness.name:
+ raise ValueError("Harness must define a non-empty 'name'")
+ if not override and harness.name in _HARNESSES:
+ raise ValueError(f"Harness {harness.name!r} is already registered")
+ _HARNESSES[harness.name] = harness
+
+
+def get_harness(name: str) -> SweTaskHarness:
+ """Look up a registered harness by name.
+
+ Args:
+ name (str): The registry key of the harness.
+
+ Returns:
+ SweTaskHarness: The registered harness.
+
+ Raises:
+ KeyError: If no harness is registered under ``name``.
+ """
+ try:
+ return _HARNESSES[name]
+ except KeyError as exc:
+ available = ", ".join(sorted(_HARNESSES)) or "(none)"
+ raise KeyError(f"Unknown SWE harness {name!r}. Registered: {available}") from exc
+
+
+def list_harnesses() -> list[str]:
+ """List the names of all registered harnesses.
+
+ Returns:
+ list[str]: The registered harness names, sorted alphabetically.
+ """
+ return sorted(_HARNESSES)
diff --git a/responses_api_agents/swe_env/requirements.txt b/responses_api_agents/swe_env/requirements.txt
new file mode 100644
index 0000000000..00ed83213e
--- /dev/null
+++ b/responses_api_agents/swe_env/requirements.txt
@@ -0,0 +1 @@
+-e nemo-gym[dev] @ ../../
diff --git a/responses_api_agents/swe_env/tests/__init__.py b/responses_api_agents/swe_env/tests/__init__.py
new file mode 100644
index 0000000000..777f2341ac
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/__init__.py
@@ -0,0 +1 @@
+"""Test suite for the swe_env agent harness."""
diff --git a/responses_api_agents/swe_env/tests/conftest.py b/responses_api_agents/swe_env/tests/conftest.py
new file mode 100644
index 0000000000..218077e1b1
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/conftest.py
@@ -0,0 +1,27 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Pytest collection guard for the swe_env tests.
+
+The flat-eval parser fixtures are recorded eval logs whose lines begin with the
+SWE-bench ``>>>>>`` sentinels. Under doctest collection those look like
+(malformed) ``>>>`` prompts, so the fixtures directory is excluded from
+collection entirely. It holds only data, never tests.
+"""
+
+from __future__ import annotations
+
+
+# Never collect anything under the fixtures tree (recorded logs / data only).
+collect_ignore_glob = ["fixtures/*"]
diff --git a/responses_api_agents/swe_env/tests/fixtures/flat_eval/apply_patch_failed.txt b/responses_api_agents/swe_env/tests/fixtures/flat_eval/apply_patch_failed.txt
new file mode 100644
index 0000000000..bb67958525
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/fixtures/flat_eval/apply_patch_failed.txt
@@ -0,0 +1,9 @@
++ cd /testbed
++ git apply -v /tmp/patch.diff
+Checking patch sphinx/ext/autodoc/__init__.py...
+error: while searching for:
+ def format_signature(self):
+error: patch failed: sphinx/ext/autodoc/__init__.py:120
+error: sphinx/ext/autodoc/__init__.py: patch does not apply
+>>>>> Patch Apply Failed
++ git checkout abc123 tests/test_ext_autodoc.py
diff --git a/responses_api_agents/swe_env/tests/fixtures/flat_eval/fallback_outside_markers.txt b/responses_api_agents/swe_env/tests/fixtures/flat_eval/fallback_outside_markers.txt
new file mode 100644
index 0000000000..bc8d678e61
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/fixtures/flat_eval/fallback_outside_markers.txt
@@ -0,0 +1,14 @@
++ cd /testbed
++ git apply -v /tmp/patch.diff
+Applied patch sphinx/ext/autodoc/__init__.py cleanly.
+>>>>> Applied Patch
++ git apply -v /tmp/test_patch.diff
+Applied patch tests/test_ext_autodoc.py cleanly.
+>>>>> Start Test Output
+============================= test session starts ==============================
+collected 3 items
+>>>>> End Test Output
+PASSED tests/test_ext_autodoc.py::test_format_signature
+PASSED tests/test_ext_autodoc.py::test_autodoc_inherited
+PASSED tests/test_ext_autodoc.py::test_autodoc_exclude_members
+=================== 3 passed in 1.92s =========================================
diff --git a/responses_api_agents/swe_env/tests/fixtures/flat_eval/no_markers.txt b/responses_api_agents/swe_env/tests/fixtures/flat_eval/no_markers.txt
new file mode 100644
index 0000000000..c4f0e56654
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/fixtures/flat_eval/no_markers.txt
@@ -0,0 +1,11 @@
++ cd /testbed
++ git apply -v /tmp/patch.diff
+Applied patch sphinx/ext/autodoc/__init__.py cleanly.
+>>>>> Applied Patch
++ git checkout abc123 tests/test_ext_autodoc.py
+Updated 1 path from the index
++ git apply -v /tmp/test_patch.diff
+error: patch failed: tests/test_ext_autodoc.py:1
+error: tests/test_ext_autodoc.py: patch does not apply
++ python -m pytest tests/test_ext_autodoc.py
+ERROR: file or directory not found: tests/test_ext_autodoc.py
diff --git a/responses_api_agents/swe_env/tests/fixtures/flat_eval/resolved_success.txt b/responses_api_agents/swe_env/tests/fixtures/flat_eval/resolved_success.txt
new file mode 100644
index 0000000000..1d0ba6a53a
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/fixtures/flat_eval/resolved_success.txt
@@ -0,0 +1,25 @@
++ source /opt/miniconda3/bin/activate
++ conda activate testbed
++ git config --global --add safe.directory /testbed
++ cd /testbed
++ git status
++ git restore .
++ git apply -v /tmp/patch.diff
+Checking patch sphinx/ext/autodoc/__init__.py...
+Applied patch sphinx/ext/autodoc/__init__.py cleanly.
+>>>>> Applied Patch
++ git checkout abc123 tests/test_ext_autodoc.py
+Updated 1 path from the index
++ git apply -v /tmp/test_patch.diff
+Checking patch tests/test_ext_autodoc.py...
+Applied patch tests/test_ext_autodoc.py cleanly.
+>>>>> Start Test Output
+============================= test session starts ==============================
+PASSED tests/test_ext_autodoc.py::test_format_signature
+PASSED tests/test_ext_autodoc.py::test_autodoc_inherited
+PASSED tests/test_ext_autodoc.py::test_autodoc_exclude_members
+SKIPPED tests/test_ext_autodoc.py::test_optional_feature
+=================== 3 passed, 1 skipped in 2.41s ===============================
+>>>>> End Test Output
++ git checkout abc123 tests/test_ext_autodoc.py
+Updated 1 path from the index
diff --git a/responses_api_agents/swe_env/tests/fixtures/flat_eval/tests_timeout.txt b/responses_api_agents/swe_env/tests/fixtures/flat_eval/tests_timeout.txt
new file mode 100644
index 0000000000..0a27e668e1
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/fixtures/flat_eval/tests_timeout.txt
@@ -0,0 +1,10 @@
++ cd /testbed
++ git apply -v /tmp/patch.diff
+Applied patch sphinx/ext/autodoc/__init__.py cleanly.
+>>>>> Applied Patch
++ git apply -v /tmp/test_patch.diff
+Applied patch tests/test_ext_autodoc.py cleanly.
+>>>>> Start Test Output
+============================= test session starts ==============================
+PASSED tests/test_ext_autodoc.py::test_autodoc_inherited
+>>>>> Tests Timed Out
diff --git a/responses_api_agents/swe_env/tests/fixtures/flat_eval/unresolved_failure.txt b/responses_api_agents/swe_env/tests/fixtures/flat_eval/unresolved_failure.txt
new file mode 100644
index 0000000000..59dc10159f
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/fixtures/flat_eval/unresolved_failure.txt
@@ -0,0 +1,16 @@
++ cd /testbed
++ git apply -v /tmp/patch.diff
+Checking patch sphinx/ext/autodoc/__init__.py...
+Applied patch sphinx/ext/autodoc/__init__.py cleanly.
+>>>>> Applied Patch
++ git apply -v /tmp/test_patch.diff
+Checking patch tests/test_ext_autodoc.py...
+Applied patch tests/test_ext_autodoc.py cleanly.
+>>>>> Start Test Output
+============================= test session starts ==============================
+FAILED tests/test_ext_autodoc.py::test_format_signature - AssertionError: signature mismatch
+PASSED tests/test_ext_autodoc.py::test_autodoc_inherited
+PASSED tests/test_ext_autodoc.py::test_autodoc_exclude_members
+=================== 2 passed, 1 failed in 2.10s ================================
+>>>>> End Test Output
++ git checkout abc123 tests/test_ext_autodoc.py
diff --git a/responses_api_agents/swe_env/tests/fixtures/swe_bench_ext/go_json.txt b/responses_api_agents/swe_env/tests/fixtures/swe_bench_ext/go_json.txt
new file mode 100644
index 0000000000..5f1200be91
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/fixtures/swe_bench_ext/go_json.txt
@@ -0,0 +1,6 @@
+{"Time":"2026-06-23T00:00:00Z","Action":"run","Package":"github.com/acme/widget","Test":"TestAlpha"}
+{"Time":"2026-06-23T00:00:00Z","Action":"pass","Package":"github.com/acme/widget","Test":"TestAlpha","Elapsed":0.01}
+{"Time":"2026-06-23T00:00:01Z","Action":"run","Package":"github.com/acme/widget","Test":"TestBeta"}
+{"Time":"2026-06-23T00:00:01Z","Action":"pass","Package":"github.com/acme/widget","Test":"TestBeta","Elapsed":0.02}
+{"Time":"2026-06-23T00:00:02Z","Action":"run","Package":"github.com/acme/widget","Test":"TestGamma"}
+{"Time":"2026-06-23T00:00:02Z","Action":"fail","Package":"github.com/acme/widget","Test":"TestGamma","Elapsed":0.01}
diff --git a/responses_api_agents/swe_env/tests/fixtures/swe_bench_ext/pytest_junit.xml b/responses_api_agents/swe_env/tests/fixtures/swe_bench_ext/pytest_junit.xml
new file mode 100644
index 0000000000..028b436db3
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/fixtures/swe_bench_ext/pytest_junit.xml
@@ -0,0 +1,10 @@
+
+
+
+
+
+
+ boom
+
+
+
diff --git a/responses_api_agents/swe_env/tests/fixtures/swe_bench_ext/pytest_text_fuzzy.txt b/responses_api_agents/swe_env/tests/fixtures/swe_bench_ext/pytest_text_fuzzy.txt
new file mode 100644
index 0000000000..d566714983
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/fixtures/swe_bench_ext/pytest_text_fuzzy.txt
@@ -0,0 +1,15 @@
+============================= test session starts ==============================
+platform linux -- Python 3.12.0, pytest-8.0.0
+collected 3 items
+
+src/pkg/tests/test_widget.py::test_alpha PASSED [ 33%]
+src/pkg/tests/test_widget.py::test_beta PASSED [ 66%]
+src/pkg/tests/test_widget.py::test_gamma FAILED [100%]
+
+=================================== FAILURES ===================================
+________________________________ test_gamma ___________________________________
+ assert 1 == 2
+E assert 1 == 2
+=========================== short test summary info ============================
+FAILED src/pkg/tests/test_widget.py::test_gamma - assert 1 == 2
+========================= 2 passed, 1 failed in 0.12s ==========================
diff --git a/responses_api_agents/swe_env/tests/test_apptainer_provider.py b/responses_api_agents/swe_env/tests/test_apptainer_provider.py
new file mode 100644
index 0000000000..d0319f4f13
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/test_apptainer_provider.py
@@ -0,0 +1,284 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Apptainer provider tests (mocked subprocess — apptainer not installed here)."""
+
+from __future__ import annotations
+
+import asyncio
+from pathlib import Path
+
+from nemo_gym.sandbox import SandboxSpec
+from responses_api_agents.swe_env.providers.apptainer_provider import ApptainerSandboxProvider
+
+
+def _patch_run(provider, scripted):
+ """Replace the provider's subprocess runner with a recording stub.
+
+ Args:
+ provider: The ApptainerSandboxProvider whose ``_run`` is patched.
+ scripted: A callable that maps the argv list to a ``(return_code, stdout,
+ stderr)`` tuple returned by the stubbed runner.
+
+ Returns:
+ A list that accumulates the argv list of each call, for later assertions.
+ """
+ calls: list[list[str]] = []
+
+ async def fake_run(*args, timeout_s=None):
+ calls.append(list(args))
+ return scripted(list(args))
+
+ provider._run = fake_run # type: ignore[assignment]
+ return calls
+
+
+def test_resolve_sif_direct_path(tmp_path: Path):
+ """A spec image that is a direct ``.sif`` path resolves to that path.
+
+ Args:
+ tmp_path: Pytest temporary directory fixture.
+ """
+ sif = tmp_path / "image.sif"
+ sif.write_text("x")
+ provider = ApptainerSandboxProvider()
+ spec = SandboxSpec(image=str(sif))
+ assert provider._resolve_sif(spec) == str(sif)
+
+
+def test_resolve_sif_glob(tmp_path: Path):
+ """A configured ``image_glob`` resolves the image against the image root.
+
+ Args:
+ tmp_path: Pytest temporary directory fixture.
+ """
+ (tmp_path / "myrepo__inst.sif").write_text("x")
+ provider = ApptainerSandboxProvider(image_root=str(tmp_path))
+ spec = SandboxSpec(image="inst", provider_options={"image_glob": "*.sif"})
+ assert provider._resolve_sif(spec).endswith("myrepo__inst.sif")
+
+
+def test_resolve_sif_fuzzy_restricts_to_sif(tmp_path: Path):
+ """The fuzzy fallback only matches ``.sif`` files, ignoring other host files.
+
+ Args:
+ tmp_path: Pytest temporary directory fixture.
+ """
+ # The fuzzy term is restricted to ``*.sif`` so an unrelated host file matching
+ # the image substring is never mistaken for a container.
+ (tmp_path / "myrepo__inst.sif").write_text("x")
+ (tmp_path / "myrepo__inst.log").write_text("not a container")
+ provider = ApptainerSandboxProvider(image_root=str(tmp_path))
+ spec = SandboxSpec(image="inst") # no explicit image_glob -> fuzzy path
+ assert provider._resolve_sif(spec).endswith("myrepo__inst.sif")
+
+
+def test_resolve_sif_fuzzy_lowercases_search_term(tmp_path: Path):
+ """A mixed-case image matches a lowercased ``.sif`` on disk via the fuzzy fallback.
+
+ Args:
+ tmp_path: Pytest temporary directory fixture.
+ """
+ (tmp_path / "repo__myinst.sif").write_text("x")
+ provider = ApptainerSandboxProvider(image_root=str(tmp_path))
+ spec = SandboxSpec(image="MyInst") # uppercase; on-disk file is lowercase
+ assert provider._resolve_sif(spec).endswith("repo__myinst.sif")
+
+
+def test_default_instance_args_restore_legacy_flags():
+ """The default launch flags include the expected apptainer exec flags."""
+ # The default launch flags are --pid and --no-mount home,tmp,bind-paths, on top
+ # of --writable-tmpfs --cleanenv. They remain overridable via the instance_args
+ # kwarg.
+ provider = ApptainerSandboxProvider()
+ assert provider._instance_args == [
+ "--writable-tmpfs",
+ "--cleanenv",
+ "--pid",
+ "--no-mount",
+ "home,tmp,bind-paths",
+ ]
+
+
+def test_instance_args_remain_overridable():
+ """An explicit ``instance_args`` value overrides the default launch flags."""
+ provider = ApptainerSandboxProvider(instance_args=["--nv"])
+ assert provider._instance_args == ["--nv"]
+ # An explicit empty list disables the defaults entirely (distinct from None).
+ assert ApptainerSandboxProvider(instance_args=[])._instance_args == []
+
+
+def test_create_issues_legacy_default_flags(tmp_path: Path):
+ """Creating an instance issues the default launch flags in the start argv.
+
+ Args:
+ tmp_path: Pytest temporary directory fixture.
+ """
+ sif = tmp_path / "image.sif"
+ sif.write_text("x")
+ provider = ApptainerSandboxProvider()
+ calls = _patch_run(provider, lambda args: (0, "out", ""))
+ asyncio.run(provider.create(SandboxSpec(image=str(sif), metadata={"instance_id": "i"})))
+ start_argv = calls[0]
+ assert "--pid" in start_argv
+ assert "--no-mount" in start_argv
+ assert "home,tmp,bind-paths" in start_argv
+
+
+def test_create_and_exec_issue_expected_argv(tmp_path: Path):
+ """Create and exec build the expected ``instance start`` and ``exec`` argv.
+
+ Args:
+ tmp_path: Pytest temporary directory fixture.
+ """
+ sif = tmp_path / "image.sif"
+ sif.write_text("x")
+ provider = ApptainerSandboxProvider()
+ calls = _patch_run(provider, lambda args: (0, "out", ""))
+
+ handle = asyncio.run(
+ provider.create(SandboxSpec(image=str(sif), workdir="/testbed", metadata={"instance_id": "i"}))
+ )
+ assert handle.provider_name == "apptainer"
+ start_argv = calls[0]
+ assert start_argv[:2] == ["instance", "start"]
+ assert str(sif) in start_argv
+
+ asyncio.run(provider.exec(handle, "echo hi", cwd="/testbed"))
+ exec_argv = calls[-1]
+ assert exec_argv[0] == "exec"
+ assert any(a.startswith("instance://") for a in exec_argv)
+ assert "--pwd" in exec_argv and "/testbed" in exec_argv
+
+
+def test_create_binds_provider_option_mounts(tmp_path: Path):
+ """Each ``provider_options['mounts']`` entry becomes a ``--bind`` argument.
+
+ Args:
+ tmp_path: Pytest temporary directory fixture.
+ """
+ sif = tmp_path / "image.sif"
+ sif.write_text("x")
+ # Real host setup dir so the canonical self-bind (src == dst) is a genuine
+ # host path and is NOT dropped by the dataset self-bind guard.
+ setup = tmp_path / "setup"
+ setup.mkdir()
+ provider = ApptainerSandboxProvider()
+ calls = _patch_run(provider, lambda args: (0, "out", ""))
+
+ spec = SandboxSpec(
+ image=str(sif),
+ workdir="/testbed",
+ metadata={"instance_id": "i"},
+ provider_options={
+ "mounts": [
+ {"src": "/host/data.jsonl", "dst": "/root/dataset/data.jsonl"},
+ {"src": str(setup), "dst": "/swebench_setup"},
+ {"src": str(setup), "dst": str(setup)},
+ {"src": "/host/ro", "dst": "/ro", "ro": True},
+ ]
+ },
+ )
+ asyncio.run(provider.create(spec))
+ start_argv = calls[0]
+ assert "--bind" in start_argv
+ binds = [start_argv[i + 1] for i, a in enumerate(start_argv) if a == "--bind"]
+ assert "/host/data.jsonl:/root/dataset/data.jsonl" in binds
+ assert f"{setup}:/swebench_setup" in binds
+ # An existing host self-bind survives unchanged.
+ assert f"{setup}:{setup}" in binds
+ # Read-only mounts get the :ro suffix (src != dst, so the guard never applies).
+ assert "/host/ro:/ro:ro" in binds
+
+
+def test_create_skips_dataset_self_bind_when_src_missing(tmp_path: Path):
+ """A self-bind whose host source does not exist is dropped from the binds.
+
+ Args:
+ tmp_path: Pytest temporary directory fixture.
+ """
+ # The dataset default mount falls back to src == dst == the in-container
+ # dataset path (not a host path) when no real dataset path is provisioned.
+ # Binding a missing host src would shadow the real dataset with an empty dir,
+ # so a self-bind whose source does not exist is skipped.
+ sif = tmp_path / "image.sif"
+ sif.write_text("x")
+ provider = ApptainerSandboxProvider()
+ calls = _patch_run(provider, lambda args: (0, "out", ""))
+
+ spec = SandboxSpec(
+ image=str(sif),
+ provider_options={
+ "mounts": [
+ # Self-bind of an in-container-only path (no host counterpart).
+ {"src": "/root/dataset/data.jsonl", "dst": "/root/dataset/data.jsonl"},
+ ]
+ },
+ )
+ asyncio.run(provider.create(spec))
+ start_argv = calls[0]
+ binds = [start_argv[i + 1] for i, a in enumerate(start_argv) if a == "--bind"]
+ # Only the scratch I/O mount survives; the phantom dataset self-bind is dropped.
+ assert len(binds) == 1
+ assert not any("data.jsonl" in b for b in binds)
+
+
+def test_create_skips_incomplete_mounts(tmp_path: Path):
+ """Mounts missing a src or dst are skipped, not emitted as half-specified binds.
+
+ Args:
+ tmp_path: Pytest temporary directory fixture.
+ """
+ sif = tmp_path / "image.sif"
+ sif.write_text("x")
+ provider = ApptainerSandboxProvider()
+ calls = _patch_run(provider, lambda args: (0, "out", ""))
+
+ spec = SandboxSpec(
+ image=str(sif),
+ provider_options={"mounts": [{"src": "/only-src"}, {"dst": "/only-dst"}, {}]},
+ )
+ asyncio.run(provider.create(spec))
+ start_argv = calls[0]
+ binds = [start_argv[i + 1] for i, a in enumerate(start_argv) if a == "--bind"]
+ # Only the scratch I/O mount survives; no half-specified binds slip through.
+ assert len(binds) == 1
+ assert not any("only-src" in b or "only-dst" in b for b in binds)
+
+
+def test_mount_binds_empty_when_no_mounts():
+ """``_mount_binds`` returns an empty list when no mounts are configured."""
+ provider = ApptainerSandboxProvider()
+ assert provider._mount_binds(SandboxSpec(image="x")) == []
+ assert provider._mount_binds(SandboxSpec(image="x", provider_options={"mounts": None})) == []
+
+
+def test_exec_timeout_returns_typed_result(tmp_path: Path):
+ """A timed-out exec returns a typed result with return code 124 and a timeout kind.
+
+ Args:
+ tmp_path: Pytest temporary directory fixture.
+ """
+ provider = ApptainerSandboxProvider()
+
+ async def timeout_run(*args, timeout_s=None):
+ raise asyncio.TimeoutError
+
+ provider._run = timeout_run # type: ignore[assignment]
+ from nemo_gym.sandbox import SandboxHandle
+
+ handle = SandboxHandle(sandbox_id="x", provider_name="apptainer", raw={"workdir": "/t", "scratch": str(tmp_path)})
+ result = asyncio.run(provider.exec(handle, "sleep 100", timeout_s=1))
+ assert result.return_code == 124
+ assert result.error_type == "timeout"
diff --git a/responses_api_agents/swe_env/tests/test_flat_eval.py b/responses_api_agents/swe_env/tests/test_flat_eval.py
new file mode 100644
index 0000000000..6142bfef27
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/test_flat_eval.py
@@ -0,0 +1,584 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests for the opt-in flat (host-graded) eval mode of the nested families.
+
+The suite has three layers:
+
+* Parser unit tests on recorded fixture logs cover the SWE-bench eval-script log
+ parser (``parse_eval_log``) on a success log, a failure log, the bad-code logs
+ (patch-apply-failed / timeout), a no-markers log, and the
+ output-outside-markers fallback. The fixtures use the
+ ``>>>>> Start/End Test Output`` shape the SWE-bench eval script emits.
+
+* Flat run_eval and grade via FakeSandbox drive the flat path of both nested
+ harnesses (``swe-bench``, ``r2e-gym``) end-to-end with a scripted provider that
+ returns a fixture log, asserting ``resolved`` is computed from ``FAIL_TO_PASS``
+ / ``PASS_TO_PASS``.
+
+* A golden-patch equivalence scaffold is skipped unless
+ ``SWE_ENV_RUN_REAL_CONTAINERS=1``. Asserting that flat ``resolved`` equals
+ nested ``resolved`` on gold patches needs apptainer, Docker, and published
+ per-instance SWE-bench ``.sif`` images; the scaffold documents the comparison
+ so it can be run on a real cluster.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import os
+from pathlib import Path
+
+import pytest
+
+from nemo_gym.sandbox import (
+ SandboxExecResult,
+ SandboxHandle,
+ SandboxStatus,
+ register_provider,
+)
+from responses_api_agents.swe_env.grading import reward_from_report
+from responses_api_agents.swe_env.harness import EvalArtifacts, SweTask
+from responses_api_agents.swe_env.harnesses import flat_eval
+from responses_api_agents.swe_env.harnesses.r2egym import R2EGymHarness
+from responses_api_agents.swe_env.harnesses.swebench import SweBenchHarness
+
+
+_FIXTURES = Path(__file__).parent / "fixtures" / "flat_eval"
+
+
+def _fixture(name: str) -> str:
+ """Read a recorded fixture log by name.
+
+ Fixtures are stored with a ``.txt`` suffix, so a caller may pass either the
+ ``.log`` stem name or the real ``.txt`` name.
+
+ Args:
+ name: The fixture file name, with either a ``.log`` or ``.txt`` suffix.
+
+ Returns:
+ The fixture file contents as text.
+ """
+ path = _FIXTURES / name
+ if not path.exists() and path.suffix == ".log":
+ path = path.with_suffix(".txt")
+ return path.read_text()
+
+
+# ---- parser: recorded fixture logs (CI) -------------------------------------
+
+
+def test_parse_success_log_all_pass():
+ """A success log parses to a status map with the expected passed and skipped tests."""
+ status_map, applied = flat_eval.parse_eval_log(_fixture("resolved_success.log"))
+ assert applied is True
+ assert status_map == {
+ "tests/test_ext_autodoc.py::test_format_signature": "PASSED",
+ "tests/test_ext_autodoc.py::test_autodoc_inherited": "PASSED",
+ "tests/test_ext_autodoc.py::test_autodoc_exclude_members": "PASSED",
+ "tests/test_ext_autodoc.py::test_optional_feature": "SKIPPED",
+ }
+ assert sorted(flat_eval.passed_tests(status_map)) == [
+ "tests/test_ext_autodoc.py::test_autodoc_exclude_members",
+ "tests/test_ext_autodoc.py::test_autodoc_inherited",
+ "tests/test_ext_autodoc.py::test_format_signature",
+ ]
+
+
+def test_parse_failure_log_strips_failed_reason():
+ """A failure log parses with the failure reason stripped down to the node id."""
+ status_map, applied = flat_eval.parse_eval_log(_fixture("unresolved_failure.log"))
+ assert applied is True
+ # The "FAILED - " line keeps only the node id.
+ assert status_map["tests/test_ext_autodoc.py::test_format_signature"] == "FAILED"
+ assert "tests/test_ext_autodoc.py::test_autodoc_inherited" in flat_eval.passed_tests(status_map)
+
+
+def test_parse_apply_patch_failed_is_untrusted():
+ """A patch-apply-failed log yields an empty status map and patch_applied False."""
+ status_map, applied = flat_eval.parse_eval_log(_fixture("apply_patch_failed.log"))
+ assert status_map == {}
+ assert applied is False
+
+
+def test_parse_timeout_is_untrusted():
+ """A timeout log yields an empty status map and patch_applied False."""
+ status_map, applied = flat_eval.parse_eval_log(_fixture("tests_timeout.log"))
+ assert status_map == {}
+ assert applied is False
+
+
+def test_parse_no_markers_is_untrusted():
+ """A log with no test-output markers yields an empty status map and patch_applied False."""
+ status_map, applied = flat_eval.parse_eval_log(_fixture("no_markers.log"))
+ assert status_map == {}
+ assert applied is False
+
+
+def test_parse_fallback_outside_markers():
+ """Per-test lines appearing after the End marker are recovered by the whole-log fallback."""
+ status_map, applied = flat_eval.parse_eval_log(_fixture("fallback_outside_markers.log"))
+ assert applied is True
+ assert len(flat_eval.passed_tests(status_map)) == 3
+
+
+def test_parse_duplicate_node_last_status_wins():
+ """For a duplicated node id the last reported status wins.
+
+ A node first reported FAILED then re-reported PASSED (e.g. via a rerun plugin)
+ ends up PASSED, and vice versa.
+ """
+ log = "\n".join(
+ [
+ flat_eval.APPLY_PATCH_PASS,
+ flat_eval.START_TEST_OUTPUT,
+ "FAILED tests/test_x.py::test_flaky",
+ "PASSED tests/test_x.py::test_flaky",
+ "PASSED tests/test_x.py::test_regressed",
+ "FAILED tests/test_x.py::test_regressed",
+ flat_eval.END_TEST_OUTPUT,
+ ]
+ )
+ status_map, applied = flat_eval.parse_eval_log(log)
+ assert applied is True
+ # Last line wins for each node, not the first.
+ assert status_map["tests/test_x.py::test_flaky"] == "PASSED"
+ assert status_map["tests/test_x.py::test_regressed"] == "FAILED"
+ assert flat_eval.passed_tests(status_map) == ["tests/test_x.py::test_flaky"]
+
+
+def test_parse_xfail_counts_as_pass():
+ """An XFAIL node counts as a passed test."""
+ log = "\n".join(
+ [
+ flat_eval.APPLY_PATCH_PASS,
+ flat_eval.START_TEST_OUTPUT,
+ "XFAIL tests/test_x.py::test_known_bug",
+ "PASSED tests/test_x.py::test_ok",
+ flat_eval.END_TEST_OUTPUT,
+ ]
+ )
+ status_map, applied = flat_eval.parse_eval_log(log)
+ assert applied is True
+ assert set(flat_eval.passed_tests(status_map)) == {
+ "tests/test_x.py::test_known_bug",
+ "tests/test_x.py::test_ok",
+ }
+
+
+# ---- flat_grade over parsed fixtures (CI) -----------------------------------
+
+
+def _task(benchmark: str = "swe-bench", **overrides) -> SweTask:
+ """Build a SweTask with sensible defaults, overridable per keyword.
+
+ Args:
+ benchmark: The benchmark name for the task.
+ **overrides: Field overrides merged onto the default task fields.
+
+ Returns:
+ A SweTask configured for the given benchmark.
+ """
+ base = dict(
+ instance_id="repo__inst-1",
+ image="img:tag",
+ base_commit="abc123",
+ repo_workdir="/testbed",
+ model_patch="diff --git a/x b/x\n",
+ fail_to_pass=["tests/test_ext_autodoc.py::test_format_signature"],
+ pass_to_pass=["tests/test_ext_autodoc.py::test_autodoc_inherited"],
+ benchmark=benchmark,
+ )
+ base.update(overrides)
+ return SweTask(**base)
+
+
+def _flat_artifacts(log: str) -> EvalArtifacts:
+ """Wrap an eval log in flat-eval EvalArtifacts.
+
+ Args:
+ log: The eval-script log text.
+
+ Returns:
+ EvalArtifacts carrying the log with a clean (non-error) flat raw payload.
+ """
+ return EvalArtifacts(test_output=log, return_code=0, patch_applied=True, raw={"error_type": None, "flat": True})
+
+
+def test_flat_grade_resolved_on_success():
+ """Flat grading resolves a success log with reward 1.0."""
+ report = flat_eval.flat_grade(_task(), _flat_artifacts(_fixture("resolved_success.log")))
+ assert report.resolved is True
+ assert report.patch_applied is True
+ assert report.patch_exists is True
+ assert reward_from_report(report) == 1.0
+
+
+def test_flat_grade_unresolved_on_failure():
+ """Flat grading leaves a failure log unresolved with reward 0.0."""
+ report = flat_eval.flat_grade(_task(), _flat_artifacts(_fixture("unresolved_failure.log")))
+ assert report.resolved is False
+ assert reward_from_report(report) == 0.0
+
+
+def test_flat_grade_unresolved_on_apply_failed():
+ """A failed patch apply grades as a legitimate unresolved, not an infra mask."""
+ report = flat_eval.flat_grade(_task(), _flat_artifacts(_fixture("apply_patch_failed.log")))
+ assert report.resolved is False
+ assert report.patch_applied is False
+ assert report.error_kind is None
+ assert reward_from_report(report) == 0.0
+
+
+# ---- consistency of flat grading --------------------------------------------
+#
+# Flat grading takes ``resolved`` straight from the parser's verdict (all F2P +
+# all P2P passed) and never re-gates it on ``patch_applied``. The parser's
+# ``log_patch_applied`` flag never changes ``resolved`` relative to a pure
+# ``compute_resolved`` verdict: whenever ``parse_eval_log`` reports
+# ``patch_applied=False`` it also returns an empty status map, so
+# ``compute_resolved`` already yields False. These tests lock in that invariant
+# so a future edit cannot reintroduce a divergent gate.
+
+
+@pytest.mark.parametrize(
+ "fixture_name",
+ [
+ "resolved_success.log",
+ "unresolved_failure.log",
+ "apply_patch_failed.log",
+ "tests_timeout.log",
+ "no_markers.log",
+ "fallback_outside_markers.log",
+ ],
+)
+def test_flat_grade_resolved_matches_ungated_compute_resolved(fixture_name):
+ """``flat_grade``'s resolved verdict agrees with a bare ``compute_resolved`` over the parsed passed-set.
+
+ The patch-applied gate is redundant and never flips the verdict True<->False.
+
+ Args:
+ fixture_name: The recorded fixture log to parse and grade.
+ """
+ from responses_api_agents.swe_env.grading import compute_resolved
+
+ task = _task()
+ log = _fixture(fixture_name)
+ status_map, _applied = flat_eval.parse_eval_log(log)
+ ungated = compute_resolved(
+ fail_to_pass=task.fail_to_pass,
+ pass_to_pass=task.pass_to_pass,
+ passed=flat_eval.passed_tests(status_map),
+ )
+ report = flat_eval.flat_grade(task, _flat_artifacts(log))
+ assert report.resolved is ungated
+
+
+@pytest.mark.parametrize(
+ "bad_code_attr",
+ ["APPLY_PATCH_FAIL", "RESET_FAILED", "TESTS_ERROR", "TESTS_TIMEOUT"],
+)
+def test_parse_eval_log_bad_code_empties_status_map_even_with_status_lines(bad_code_attr):
+ """A bad code forces an empty status map and patch_applied False even with per-test status lines.
+
+ This is what makes the flat_grade patch-applied gate redundant: no path yields
+ patch_applied=False together with a non-empty status map.
+
+ Args:
+ bad_code_attr: Name of the bad-code marker attribute on ``flat_eval``.
+ """
+ bad_code = getattr(flat_eval, bad_code_attr)
+ log = "\n".join(
+ [
+ bad_code,
+ flat_eval.START_TEST_OUTPUT,
+ "PASSED tests/test_ext_autodoc.py::test_format_signature",
+ "PASSED tests/test_ext_autodoc.py::test_autodoc_inherited",
+ flat_eval.END_TEST_OUTPUT,
+ ]
+ )
+ status_map, applied = flat_eval.parse_eval_log(log)
+ assert applied is False
+ assert status_map == {}
+ # And it grades as a legitimate unresolved (not an infra mask): error_kind
+ # stays None, resolved False -> reward 0.0, matching the flat families.
+ report = flat_eval.flat_grade(_task(), _flat_artifacts(log))
+ assert report.resolved is False
+ assert report.error_kind is None
+ assert reward_from_report(report) == 0.0
+
+
+def test_flat_grade_resolved_does_not_gate_on_artifact_patch_applied():
+ """Flat ``resolved`` is the parser's verdict only and ignores the artifact's patch_applied flag.
+
+ Even if the EvalArtifacts carries patch_applied False (e.g. the model patch
+ did not cleanly apply), a passing eval log still resolves, since grading is
+ based on the tests rather than the apply status.
+ """
+ artifacts = EvalArtifacts(
+ test_output=_fixture("resolved_success.log"),
+ return_code=0,
+ patch_applied=False,
+ raw={"error_type": None, "flat": True},
+ )
+ report = flat_eval.flat_grade(_task(), artifacts)
+ assert report.resolved is True
+ assert reward_from_report(report) == 1.0
+
+
+def test_flat_grade_masks_infra_error():
+ """Flat grading masks an infra timeout to reward 0.0 with a timeout error kind."""
+ artifacts = EvalArtifacts(test_output="", return_code=1, raw={"error_type": "timeout", "flat": True})
+ report = flat_eval.flat_grade(_task(), artifacts)
+ assert report.error_kind == "timeout"
+ assert reward_from_report(report) == 0.0
+
+
+def test_flat_grade_masks_missing_eval_script():
+ """Flat grading masks a missing eval script to reward 0.0 with an eval_error kind."""
+ artifacts = EvalArtifacts(test_output="", return_code=1, raw={"error_type": "eval_error", "flat": True})
+ report = flat_eval.flat_grade(_task(), artifacts)
+ assert report.error_kind == "eval_error"
+ assert reward_from_report(report) == 0.0
+
+
+# ---- gating (CI) ------------------------------------------------------------
+
+
+def test_flat_eval_enabled_harness_flag():
+ """The harness-level flat-eval flag enables flat eval."""
+ assert flat_eval.flat_eval_enabled(True, _task()) is True
+
+
+def test_flat_eval_enabled_task_metadata():
+ """Per-task ``flat_eval`` metadata enables flat eval."""
+ assert flat_eval.flat_eval_enabled(False, _task(metadata={"flat_eval": True})) is True
+
+
+def test_flat_eval_disabled_by_default():
+ """Flat eval is disabled when neither the harness flag nor task metadata enables it."""
+ assert flat_eval.flat_eval_enabled(False, _task()) is False
+
+
+def test_swebench_supports_provider_gating():
+ """The swe-bench harness allows apptainer only when nested, and any exec provider when flat."""
+ # Default (nested): apptainer only.
+ nested = SweBenchHarness("swe-bench")
+ assert nested.supports_provider("apptainer") is True
+ assert nested.supports_provider("docker") is False
+ assert nested.supports_provider("opensandbox") is False
+ # Flat-capable instance: any exec provider.
+ flat = SweBenchHarness("swe-bench", flat_eval=True)
+ assert flat.supports_provider("docker") is True
+ assert flat.supports_provider("opensandbox") is True
+ assert flat.grade_strategy == "flat-host-grade"
+
+
+def test_r2egym_supports_provider_gating():
+ """The r2e-gym harness allows apptainer only when nested, and any exec provider when flat."""
+ nested = R2EGymHarness()
+ assert nested.supports_provider("apptainer") is True
+ assert nested.supports_provider("docker") is False
+ flat = R2EGymHarness(flat_eval=True)
+ assert flat.supports_provider("docker") is True
+ assert flat.supports_provider("opensandbox") is True
+ assert flat.grade_strategy == "flat-host-grade"
+
+
+# ---- flat run_eval end-to-end via FakeSandbox (CI) --------------------------
+
+
+class _FakeFlatProvider:
+ """Scripted provider: ``bash eval.sh ...`` streams a fixture log; ``cat`` echoes it."""
+
+ name = "fake-flat-eval"
+
+ def __init__(self, *, log_text="", run_rc=0, error_type=None, stream_empty=False, **_):
+ """Configure the scripted flat-eval provider's responses.
+
+ Args:
+ log_text: The eval-script log text returned by the run and ``cat``.
+ run_rc: Return code returned for the eval-script run.
+ error_type: Optional error type attached to the run result.
+ stream_empty: When True, the eval-script run streams empty stdout so
+ the harness falls back to reading the tee'd log file.
+ **_: Ignored extra keyword arguments.
+ """
+ self._log_text = log_text
+ self._run_rc = run_rc
+ self._error_type = error_type
+ self._stream_empty = stream_empty
+ self.commands: list[str] = []
+ self.uploaded: dict[str, str] = {}
+
+ async def create(self, spec):
+ return SandboxHandle(sandbox_id="fake", provider_name=self.name, raw={"workdir": spec.workdir})
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ self.commands.append(command)
+ if command.startswith("cat "):
+ return SandboxExecResult(stdout=self._log_text, stderr="", return_code=0)
+ # The eval script run.
+ stdout = "" if self._stream_empty else self._log_text
+ return SandboxExecResult(stdout=stdout, stderr="", return_code=self._run_rc, error_type=self._error_type)
+
+ async def upload_file(self, handle, local_path, remote_path):
+ try:
+ with open(local_path, encoding="utf-8") as fh:
+ self.uploaded[remote_path] = fh.read()
+ except OSError:
+ self.uploaded[remote_path] = ""
+ return None
+
+ async def download_file(self, *a, **k):
+ return None
+
+ async def status(self, handle):
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ return None
+
+ async def aclose(self):
+ return None
+
+
+register_provider("fake-flat-eval", _FakeFlatProvider, override=True)
+
+
+def _drive_flat(harness, task, *, log_text, run_rc=0, error_type=None, stream_empty=False):
+ """Drive materialize -> run_eval -> grade for a flat harness via the scripted provider.
+
+ Args:
+ harness: The flat-capable harness under test.
+ task: The SweTask to evaluate.
+ log_text: The eval-script log text the provider returns.
+ run_rc: Return code returned for the eval-script run.
+ error_type: Optional error type attached to the run result.
+ stream_empty: When True, the run streams empty stdout so the harness falls
+ back to reading the tee'd log file.
+
+ Returns:
+ A tuple of the graded report, the EvalArtifacts, and the provider instance.
+ """
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ async def _go():
+ provider = {
+ "fake-flat-eval": {
+ "log_text": log_text,
+ "run_rc": run_rc,
+ "error_type": error_type,
+ "stream_empty": stream_empty,
+ }
+ }
+ env = await AsyncSweEnvironment.start(provider, harness.build_spec(task))
+ try:
+ await harness.materialize(env, task)
+ artifacts = await harness.run_eval(env, task)
+ return harness.grade(task, artifacts), artifacts, env.sandbox._provider
+ finally:
+ await env.cleanup()
+
+ return asyncio.run(_go())
+
+
+def test_swebench_flat_run_eval_resolved():
+ """The swe-bench flat path resolves a success run and uploads the eval script."""
+ harness = SweBenchHarness("swe-bench", flat_eval=True)
+ task = _task(metadata={"eval_script": "echo running", "flat_eval": True})
+ report, artifacts, provider = _drive_flat(harness, task, log_text=_fixture("resolved_success.log"))
+ assert artifacts.raw["flat"] is True
+ assert report.resolved is True
+ assert reward_from_report(report) == 1.0
+ # The eval script was uploaded into the sandbox.
+ assert provider.uploaded.get(flat_eval.EVAL_SCRIPT_PATH, "").startswith("echo running")
+
+
+def test_swebench_flat_run_eval_unresolved():
+ """The swe-bench flat path leaves a failure run unresolved."""
+ harness = SweBenchHarness("swe-bench", flat_eval=True)
+ task = _task(metadata={"eval_script": "echo running"})
+ report, _artifacts, _ = _drive_flat(harness, task, log_text=_fixture("unresolved_failure.log"))
+ assert report.resolved is False
+
+
+def test_swebench_flat_run_eval_stream_empty_uses_log_file():
+ """When streamed output is empty, run_eval reads back the tee'd log file."""
+ harness = SweBenchHarness("swe-bench", flat_eval=True)
+ task = _task(metadata={"eval_script": "echo running"})
+ report, _artifacts, provider = _drive_flat(
+ harness, task, log_text=_fixture("resolved_success.log"), stream_empty=True
+ )
+ assert any(cmd.startswith("cat ") for cmd in provider.commands)
+ assert report.resolved is True
+
+
+def test_swebench_flat_run_eval_masks_sandbox_error():
+ """The swe-bench flat path masks a sandbox error reported by the run."""
+ harness = SweBenchHarness("swe-bench", flat_eval=True)
+ task = _task(metadata={"eval_script": "echo running"})
+ report, artifacts, _ = _drive_flat(harness, task, log_text="", run_rc=1, error_type="sandbox")
+ assert artifacts.raw["error_type"] == "sandbox"
+ assert report.error_kind == "sandbox"
+
+
+def test_swebench_flat_run_eval_missing_script_masks_eval_error():
+ """A missing eval script is masked as an eval_error."""
+ harness = SweBenchHarness("swe-bench", flat_eval=True)
+ task = _task(metadata={}) # no eval_script
+ report, artifacts, _ = _drive_flat(harness, task, log_text="")
+ assert artifacts.raw["error_type"] == "eval_error"
+ assert report.error_kind == "eval_error"
+
+
+def test_r2egym_flat_run_eval_resolved_via_task_metadata():
+ """Per-task ``flat_eval`` metadata drives the r2e-gym flat path to a resolved run."""
+ harness = R2EGymHarness(flat_eval=True)
+ task = _task(benchmark="r2e-gym", instance_id="r2e__pkg-1", metadata={"eval_script": "echo run"})
+ report, artifacts, _ = _drive_flat(harness, task, log_text=_fixture("resolved_success.log"))
+ assert artifacts.raw["flat"] is True
+ assert report.resolved is True
+
+
+# ---- infra-gated golden-patch equivalence scaffold --------------------------
+
+
+@pytest.mark.skipif(
+ os.environ.get("SWE_ENV_RUN_REAL_CONTAINERS") != "1",
+ reason=(
+ "Real flat-vs-nested equivalence needs apptainer + Docker + published per-instance "
+ "SWE-bench .sif images, which are not available in CI/this workstation. "
+ "Set SWE_ENV_RUN_REAL_CONTAINERS=1 on a cluster that has them."
+ ),
+)
+def test_flat_vs_nested_equivalence_on_gold(): # pragma: no cover - infra-gated
+ """Scaffold asserting flat ``resolved`` equals nested ``resolved`` on gold patches.
+
+ On a real cluster this would, for a small set of instances with their gold
+ ``model_patch``:
+
+ 1. Run the nested path (apptainer): ``SweBenchHarness("swe-bench")`` with
+ ``run_local_evaluation`` to get the nested ``resolved`` verdict.
+ 2. Run the flat path (docker/apptainer):
+ ``SweBenchHarness("swe-bench", flat_eval=True)`` with the SWE-bench
+ ``make_test_spec(instance).eval_script`` to get the flat ``resolved``
+ verdict.
+ 3. Assert ``flat_report.resolved == nested_report.resolved`` for every
+ instance (gold patches must resolve under both graders).
+
+ The dataset, .sif images, and both runtimes are provisioned out of band, which
+ is why this is infra-gated.
+ """
+ raise AssertionError("equivalence harness must be implemented against a real cluster")
diff --git a/responses_api_agents/swe_env/tests/test_lifecycle.py b/responses_api_agents/swe_env/tests/test_lifecycle.py
new file mode 100644
index 0000000000..585be59ecb
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/test_lifecycle.py
@@ -0,0 +1,164 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Sandbox lifecycle (``acquire_sandbox``) and ``verify_task`` happy/timeout/empty paths.
+
+These tests cover always-teardown on context exit and the fresh-sandbox verify
+sequence, including the resolved, empty-patch fast path, and eval-timeout cases.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+import pytest
+
+import responses_api_agents.swe_env.harnesses # noqa: F401 (register harnesses)
+from nemo_gym.sandbox import SandboxExecResult, SandboxHandle, SandboxStatus
+from resources_servers.swe_env.verify_task import verify_task
+from responses_api_agents.swe_env.harness import SweTask
+from responses_api_agents.swe_env.harnesses.swe_bench_ext import SweBenchExtHarness
+from responses_api_agents.swe_env.lifecycle import acquire_sandbox
+
+
+class _CountingProvider:
+ """Provider instance passed directly so the test can count create/close/exec.
+
+ Args:
+ exec_sleep: Seconds to sleep inside each ``exec`` call, used to simulate a
+ slow evaluation that triggers the eval timeout.
+ test_output: Stdout returned for pytest commands. The trailing-status
+ pytest format is the shape the test parser recognizes, and the ``.py``
+ path normalizes to the F2P id in ``_task``.
+ """
+
+ name = "fake-life"
+
+ def __init__(self, *, exec_sleep=0.0, test_output="tests/test_x.py::a PASSED\n"):
+ self.create_count = 0
+ self.close_count = 0
+ self._exec_sleep = exec_sleep
+ self._test_output = test_output
+
+ async def create(self, spec):
+ self.create_count += 1
+ return SandboxHandle(
+ sandbox_id=f"sb-{self.create_count}", provider_name=self.name, raw={"workdir": spec.workdir}
+ )
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ if self._exec_sleep:
+ await asyncio.sleep(self._exec_sleep)
+ if "pytest" in command:
+ return SandboxExecResult(stdout=self._test_output, stderr="", return_code=0)
+ return SandboxExecResult(stdout="", stderr="", return_code=0)
+
+ async def upload_file(self, *a, **k):
+ return None
+
+ async def download_file(self, *a, **k):
+ return None
+
+ async def status(self, handle):
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ self.close_count += 1
+
+ async def aclose(self):
+ return None
+
+
+def _task(**kw) -> SweTask:
+ """Build a SweTask with sensible defaults, overridable per keyword.
+
+ Args:
+ **kw: Field overrides merged onto the default task fields.
+
+ Returns:
+ A SweTask configured for the swe-bench-ext benchmark.
+ """
+ base = dict(
+ instance_id="inst-1",
+ image="img:tag",
+ base_commit="HEAD",
+ test_command="python -m pytest -rA -q",
+ model_patch="diff --git a/x b/x\n",
+ test_framework="pytest",
+ fail_to_pass=["tests/test_x.py::a"],
+ benchmark="swe-bench-ext",
+ )
+ base.update(kw)
+ return SweTask(**base)
+
+
+# ---- acquire_sandbox: starts an env, ALWAYS stops it ------------------------
+
+
+def test_acquire_sandbox_starts_and_cleans_up():
+ """``acquire_sandbox`` creates one sandbox and tears it down on normal exit."""
+ provider = _CountingProvider()
+
+ async def run():
+ spec = SweBenchExtHarness().build_spec(_task())
+ async with acquire_sandbox(provider, spec, instance_id="inst-1") as env:
+ assert env.sandbox_id is not None
+ return provider.create_count, provider.close_count
+
+ created, closed = asyncio.run(run())
+ assert created == 1
+ assert closed == 1 # torn down on normal exit
+
+
+def test_acquire_sandbox_cleans_up_on_exception():
+ """``acquire_sandbox`` tears down the sandbox even when the body raises."""
+ provider = _CountingProvider()
+
+ async def run():
+ spec = SweBenchExtHarness().build_spec(_task())
+ with pytest.raises(RuntimeError):
+ async with acquire_sandbox(provider, spec) as env:
+ assert env.sandbox_id is not None
+ raise RuntimeError("boom")
+
+ asyncio.run(run())
+ assert provider.close_count == 1 # torn down even on exception
+
+
+# ---- verify_task: resolved / empty-patch fast path / eval-timeout mask -------
+
+
+def test_verify_task_resolved_in_fresh_sandbox():
+ """``verify_task`` resolves a passing task in a freshly created sandbox."""
+ provider = _CountingProvider()
+ report = asyncio.run(verify_task(provider, _task()))
+ assert report.resolved is True
+ assert provider.create_count == 1
+ assert provider.close_count == 1
+
+
+def test_verify_task_empty_patch_fast_path_no_create():
+ """An empty model patch short-circuits to unresolved without creating a sandbox."""
+ provider = _CountingProvider()
+ report = asyncio.run(verify_task(provider, _task(model_patch="")))
+ assert report.patch_exists is False
+ assert report.resolved is False
+ assert provider.create_count == 0 # no sandbox spun up for an empty patch
+
+
+def test_verify_task_eval_timeout_masks():
+ """An evaluation that exceeds the eval timeout is masked as an eval_timeout error."""
+ provider = _CountingProvider(exec_sleep=0.5)
+ report = asyncio.run(verify_task(provider, _task(), eval_timeout_s=0.05))
+ assert report.error_kind == "eval_timeout"
diff --git a/responses_api_agents/swe_env/tests/test_model_endpoint.py b/responses_api_agents/swe_env/tests/test_model_endpoint.py
new file mode 100644
index 0000000000..dd38e324eb
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/test_model_endpoint.py
@@ -0,0 +1,57 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for the model-server egress primitive that resolves a model endpoint per provider."""
+
+from __future__ import annotations
+
+import pytest
+
+from responses_api_agents.swe_env.model_endpoint import ModelEgressUnavailable, ModelEndpoint, resolve
+
+
+def test_apptainer_uses_host_loopback_by_default():
+ """Apptainer resolves to the host loopback base URL when none is configured."""
+ ep = resolve("apptainer", {"model": "qwen"})
+ assert ep.base_url == "http://127.0.0.1:8000/v1"
+ assert ep.model == "qwen"
+
+
+def test_docker_uses_configured_base_when_present():
+ """Docker uses the explicitly configured base URL."""
+ ep = resolve("docker", {"base_url": "http://10.0.0.5:8000/v1"})
+ assert ep.base_url == "http://10.0.0.5:8000/v1"
+
+
+def test_opensandbox_requires_service_url():
+ """Opensandbox raises when no reachable service URL is supplied."""
+ with pytest.raises(ModelEgressUnavailable):
+ resolve("opensandbox", {"base_url": "http://127.0.0.1:8000/v1"})
+
+
+def test_opensandbox_with_service_url_ok():
+ """Opensandbox resolves to the provided service URL."""
+ ep = resolve("opensandbox", {"model": "m"}, opensandbox_service_url="http://gym-model.svc.cluster.local/v1")
+ assert ep.base_url == "http://gym-model.svc.cluster.local/v1"
+
+
+def test_to_sandbox_env_is_minimal():
+ """The sandbox env carries only the base URL, API key, and model name."""
+ ak_value = "abc-test"
+ env = ModelEndpoint(base_url="http://h/v1", api_key=ak_value, model="m").to_sandbox_env()
+ assert env["OPENAI_BASE_URL"] == "http://h/v1"
+ assert env["OPENAI_API_KEY"] == ak_value
+ assert env["NEMO_GYM_MODEL"] == "m"
+ # never leaks a full global-config dict
+ assert "NEMO_GYM_CONFIG_DICT" not in env
diff --git a/responses_api_agents/swe_env/tests/test_nv_internal.py b/responses_api_agents/swe_env/tests/test_nv_internal.py
new file mode 100644
index 0000000000..a5e6851f39
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/test_nv_internal.py
@@ -0,0 +1,548 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests for the nv-internal-1 harness, driven by a FakeSandbox provider.
+
+nv-internal-1 is flat + host-graded, so it runs on any exec-capable provider.
+The scripted provider returns the parsing_script ``output.json`` report on the
+``cat /root/output.json`` hop; grading is a pure host-side parse.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+
+from nemo_gym.sandbox import (
+ SandboxExecResult,
+ SandboxHandle,
+ SandboxStatus,
+ register_provider,
+)
+from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+from responses_api_agents.swe_env.grading import reward_from_report
+from responses_api_agents.swe_env.harness import EvalArtifacts, SweEvalReport, SweTask
+from responses_api_agents.swe_env.harnesses.nv_internal import (
+ NV_DEFAULT_WORKDIR,
+ NVInternalHarness,
+ _coerce_test_list,
+ _format_test_files,
+ _nv_workdir,
+ _parse_dockerfile_env,
+ _resolve_required_tests,
+ parse_passed_tests,
+)
+
+
+class _FakeProvider:
+ """Scripted provider: ``cat /root/output.json`` returns a canned report."""
+
+ name = "fake-nv"
+
+ def __init__(self, *, report="", apply_rc=0, **_):
+ """Configure the scripted provider's responses.
+
+ Args:
+ report: JSON report stdout returned for ``cat /root/output.json``.
+ apply_rc: Return code returned for ``git apply`` commands.
+ **_: Ignored extra keyword arguments.
+ """
+ self._report = report
+ self._apply_rc = apply_rc
+
+ async def create(self, spec):
+ return SandboxHandle(sandbox_id="fake", provider_name=self.name, raw={"workdir": spec.workdir})
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ if "cat /root/output.json" in command:
+ return SandboxExecResult(stdout=self._report, stderr="", return_code=0)
+ if "git apply" in command:
+ return SandboxExecResult(stdout="", stderr="", return_code=self._apply_rc)
+ return SandboxExecResult(stdout="", stderr="", return_code=0)
+
+ async def upload_file(self, *a, **k):
+ return None
+
+ async def download_file(self, *a, **k):
+ return None
+
+ async def status(self, handle):
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ return None
+
+ async def aclose(self):
+ return None
+
+
+register_provider("fake-nv", _FakeProvider, override=True)
+
+
+class _RecordingProvider:
+ """Provider that records exec ``cwd`` per command and captures uploads.
+
+ Uploads are captured as ``{target_path: content}`` by reading the temp file
+ that ``write_text`` hands to ``upload_file``; execs are captured as a list of
+ ``(command, cwd)`` so tests can assert which directory each hop ran in.
+ """
+
+ name = "fake-nv-rec"
+
+ def __init__(self, *, report="", **_):
+ """Configure the recording provider's canned report.
+
+ Args:
+ report: JSON report stdout returned for ``cat /root/output.json``.
+ **_: Ignored extra keyword arguments.
+ """
+ self._report = report
+ self.execs: list[tuple[str, str | None]] = []
+ self.uploads: dict[str, str] = {}
+
+ async def create(self, spec):
+ return SandboxHandle(sandbox_id="fake", provider_name=self.name, raw={"workdir": spec.workdir})
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ self.execs.append((command, cwd))
+ if "cat /root/output.json" in command:
+ return SandboxExecResult(stdout=self._report, stderr="", return_code=0)
+ return SandboxExecResult(stdout="", stderr="", return_code=0)
+
+ async def upload_file(self, handle, source_path, target_path):
+ with open(source_path, encoding="utf-8") as fh:
+ self.uploads[target_path] = fh.read()
+
+ async def download_file(self, *a, **k):
+ return None
+
+ async def status(self, handle):
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ return None
+
+ async def aclose(self):
+ return None
+
+
+register_provider("fake-nv-rec", _RecordingProvider, override=True)
+
+
+def _task(**overrides) -> SweTask:
+ """Build an nv-internal-1 SweTask with sensible defaults, overridable per keyword.
+
+ Args:
+ **overrides: Field overrides merged onto the default task fields.
+
+ Returns:
+ A SweTask configured for the nv-internal-1 benchmark.
+ """
+ base = dict(
+ instance_id="nv-inst-1",
+ image="img:tag",
+ base_commit="abc123",
+ repo_workdir="/app",
+ model_patch="diff --git a/x b/x\n",
+ fail_to_pass=["pkg/test_x.py::a"],
+ pass_to_pass=["pkg/test_x.py::b"],
+ benchmark="nv-internal-1",
+ metadata={
+ "run_script": "echo run\n",
+ "parsing_script": "import sys\n",
+ "selected_test_files_to_run": ["pkg/test_x.py"],
+ },
+ )
+ base.update(overrides)
+ return SweTask(**base)
+
+
+def _report(*passed, failed=()):
+ """Build a JSON test report with the given passed and failed test names.
+
+ Args:
+ *passed: Names of tests reported as PASSED.
+ failed: Names of tests reported as FAILED.
+
+ Returns:
+ The report serialized as a JSON string under a ``tests`` key.
+ """
+ tests = [{"name": name, "status": "PASSED"} for name in passed]
+ tests += [{"name": name, "status": "FAILED"} for name in failed]
+ return json.dumps({"tests": tests})
+
+
+async def _run(provider_cfg, task) -> SweEvalReport:
+ """Drive reset -> materialize -> run_eval -> grade against a scripted provider.
+
+ Args:
+ provider_cfg: Provider configuration mapping for the ``fake-nv`` provider.
+ task: The SweTask to evaluate.
+
+ Returns:
+ The graded SweEvalReport for the run.
+ """
+ harness = NVInternalHarness()
+ env = await AsyncSweEnvironment.start({"fake-nv": provider_cfg}, harness.build_spec(task))
+ try:
+ await harness.reset_repo(env, task)
+ await harness.materialize(env, task)
+ artifacts = await harness.run_eval(env, task)
+ finally:
+ await env.cleanup()
+ return harness.grade(task, artifacts)
+
+
+# ---- pure helpers -----------------------------------------------------------
+
+
+def test_parse_passed_tests():
+ """``parse_passed_tests`` returns only PASSED names and ignores malformed entries."""
+ report = {"tests": [{"name": "a", "status": "PASSED"}, {"name": "b", "status": "FAILED"}]}
+ assert parse_passed_tests(report) == ["a"]
+ assert parse_passed_tests({}) == []
+ # Malformed entries are ignored, not crashed on.
+ assert parse_passed_tests({"tests": ["junk", {"status": "PASSED"}]}) == []
+
+
+def test_format_test_files():
+ """``_format_test_files`` joins list/JSON/CSV inputs into a comma-separated string."""
+ assert _format_test_files(["a", "b"]) == "a,b"
+ assert _format_test_files('["a", "b"]') == "a,b"
+ assert _format_test_files("a,b") == "a,b"
+ assert _format_test_files(None) == ""
+
+
+def test_format_test_files_single_quoted_list():
+ """``_format_test_files`` parses repr-style single-quoted lists.
+
+ Single-quoted lists are not valid JSON, so they are parsed with
+ ``ast.literal_eval``; unparseable bracketed text falls back to the raw string.
+ """
+ assert _format_test_files("['pkg/test_x.py', 'pkg/test_y.py']") == "pkg/test_x.py,pkg/test_y.py"
+ # A single-element single-quoted list.
+ assert _format_test_files("['only.py']") == "only.py"
+ # Unparseable bracketed text falls back to the raw string, not a crash.
+ assert _format_test_files("[not a list") == "[not a list"
+
+
+def test_build_spec():
+ """The nv-internal-1 harness builds a sandbox spec from a task."""
+ harness = NVInternalHarness()
+ assert harness.name == "nv-internal-1"
+ assert harness.grade_strategy == "flat-host-grade"
+ spec = harness.build_spec(_task())
+ assert spec.image == "img:tag"
+ assert spec.workdir == "/app"
+ assert spec.metadata["instance_id"] == "nv-inst-1"
+
+
+def test_supports_any_provider():
+ """The nv-internal-1 harness supports any exec-capable provider."""
+ assert NVInternalHarness().supports_provider("docker") is True
+ assert NVInternalHarness().supports_provider("apptainer") is True
+
+
+def test_grade_masks_on_infra_error():
+ """Grading masks an infra timeout to reward 0.0 and records its error kind."""
+ harness = NVInternalHarness()
+ report = harness.grade(_task(), EvalArtifacts(test_output="", return_code=1, raw={"error_type": "timeout"}))
+ assert report.error_kind == "timeout"
+ assert reward_from_report(report) == 0.0
+
+
+def test_grade_masks_on_sandbox_error():
+ """Grading masks a sandbox error to reward 0.0 and records its error kind."""
+ harness = NVInternalHarness()
+ report = harness.grade(_task(), EvalArtifacts(test_output="", return_code=1, raw={"error_type": "sandbox"}))
+ assert report.error_kind == "sandbox"
+ assert reward_from_report(report) == 0.0
+
+
+def test_grade_empty_report_is_unresolved():
+ """An empty report grades as unresolved."""
+ harness = NVInternalHarness()
+ report = harness.grade(_task(), EvalArtifacts(test_output="", return_code=0, patch_applied=True))
+ assert report.resolved is False
+
+
+def test_grade_malformed_report_is_unresolved():
+ """A malformed (non-JSON) report grades as unresolved."""
+ harness = NVInternalHarness()
+ report = harness.grade(_task(), EvalArtifacts(test_output="not json", return_code=0, patch_applied=True))
+ assert report.resolved is False
+
+
+# ---- full reset -> materialize -> run_eval -> grade -------------------------
+
+
+def test_resolved():
+ """A run with all required tests passing resolves with reward 1.0."""
+ report = _report("pkg/test_x.py::a", "pkg/test_x.py::b")
+ result = asyncio.run(_run({"report": report}, _task()))
+ assert result.patch_applied is True
+ assert result.resolved is True
+ assert reward_from_report(result) == 1.0
+
+
+def test_unresolved_failing_required_test():
+ """A failing fail-to-pass test leaves the run unresolved with reward 0.0."""
+ report = _report("pkg/test_x.py::b", failed=["pkg/test_x.py::a"])
+ result = asyncio.run(_run({"report": report}, _task()))
+ assert result.resolved is False
+ assert reward_from_report(result) == 0.0
+
+
+def test_unresolved_missing_required_test():
+ """A required test missing from the report leaves the run unresolved."""
+ report = _report("pkg/test_x.py::a")
+ result = asyncio.run(_run({"report": report}, _task()))
+ assert result.resolved is False
+
+
+def test_patch_apply_rc_does_not_gate_resolved():
+ """A non-zero patch-apply return code does not gate ``resolved``.
+
+ Grading derives ``resolved`` from the tests alone, so a rejected patch
+ (apply_rc != 0) with all required tests passing is still resolved.
+ """
+ report = _report("pkg/test_x.py::a", "pkg/test_x.py::b")
+ result = asyncio.run(_run({"report": report, "apply_rc": 1}, _task()))
+ assert result.patch_applied is False
+ assert result.resolved is True
+ assert reward_from_report(result) == 1.0
+
+
+# ---- *_select precedence ----------------------------------------------------
+
+
+def test_resolve_required_tests_prefers_select_keys():
+ """``fail_to_pass_select`` / ``pass_to_pass_select`` take precedence over the plain keys."""
+ task = _task(
+ fail_to_pass=["plain::f2p"],
+ pass_to_pass=["plain::p2p"],
+ metadata={
+ "fail_to_pass_select": ["sel::f2p"],
+ "pass_to_pass_select": ["sel::p2p"],
+ },
+ )
+ f2p, p2p = _resolve_required_tests(task)
+ assert f2p == ["sel::f2p"]
+ assert p2p == ["sel::p2p"]
+
+
+def test_resolve_required_tests_falls_back_to_plain_keys():
+ """Without ``*_select`` keys, the plain fail_to_pass / pass_to_pass keys are used."""
+ task = _task(fail_to_pass=["plain::f2p"], pass_to_pass=["plain::p2p"], metadata={})
+ f2p, p2p = _resolve_required_tests(task)
+ assert f2p == ["plain::f2p"]
+ assert p2p == ["plain::p2p"]
+
+
+def test_resolve_required_tests_parses_stringified_select():
+ """A ``*_select`` value given as a repr-style stringified list is parsed."""
+ task = _task(
+ metadata={
+ "fail_to_pass_select": "['sel::f2p']",
+ "pass_to_pass_select": "['sel::p2p']",
+ },
+ )
+ f2p, p2p = _resolve_required_tests(task)
+ assert f2p == ["sel::f2p"]
+ assert p2p == ["sel::p2p"]
+
+
+def test_coerce_test_list():
+ """``_coerce_test_list`` accepts lists and stringified lists, returning [] on bad input."""
+ assert _coerce_test_list(["a", "b"]) == ["a", "b"]
+ assert _coerce_test_list("['a', 'b']") == ["a", "b"]
+ assert _coerce_test_list('["a", "b"]') == ["a", "b"]
+ assert _coerce_test_list("not a list") == []
+ assert _coerce_test_list("[broken") == []
+
+
+def test_resolved_uses_select_tests_end_to_end():
+ """End to end, ``*_select`` precedence resolves a run whose report has only the select tests."""
+ # The report only contains the *_select tests; the plain keys would be unmet.
+ report = _report("sel::f2p", "sel::p2p")
+ task = _task(
+ fail_to_pass=["plain::f2p"],
+ pass_to_pass=["plain::p2p"],
+ metadata={
+ "run_script": "echo run\n",
+ "parsing_script": "import sys\n",
+ "selected_test_files_to_run": ["pkg/test_x.py"],
+ "fail_to_pass_select": ["sel::f2p"],
+ "pass_to_pass_select": ["sel::p2p"],
+ },
+ )
+ result = asyncio.run(_run({"report": report}, task))
+ assert result.resolved is True
+
+
+# ---- dockerfile ENV replay --------------------------------------------------
+
+
+def test_parse_dockerfile_env_equals_and_space_forms():
+ """``_parse_dockerfile_env`` parses both ``ENV K=V`` and ``ENV K V`` forms, skipping non-ENV lines."""
+ task = _task(
+ metadata={
+ "base_dockerfile": "FROM ubuntu\nENV FOO=bar\nENV SPACED spaced_value\n",
+ "instance_dockerfile": "ENV BAZ = qux\nRUN echo hi\n",
+ },
+ )
+ env = _parse_dockerfile_env(task)
+ assert env["FOO"] == "bar"
+ assert env["SPACED"] == "spaced_value"
+ assert env["BAZ"] == "qux"
+ assert "RUN" not in env
+
+
+def test_parse_dockerfile_env_absent_is_noop():
+ """``_parse_dockerfile_env`` returns an empty mapping when no dockerfile is present."""
+ assert _parse_dockerfile_env(_task(metadata={})) == {}
+
+
+def test_build_spec_injects_dockerfile_env():
+ """``build_spec`` injects dockerfile ENV entries while preserving the existing git env."""
+ task = _task(metadata={"base_dockerfile": "ENV PATH=/custom/bin:$PATH\n"})
+ spec = NVInternalHarness().build_spec(task)
+ # Existing git env preserved; dockerfile ENV injected.
+ assert spec.env["GIT_CONFIG_GLOBAL"] == "/dev/null"
+ assert spec.env["PATH"] == "/custom/bin:$PATH"
+
+
+# ---- dotted script keys are uploaded ----------------------------------------
+
+
+async def _run_recording(task) -> _RecordingProvider:
+ """Drive reset -> materialize -> run_eval with a recording provider.
+
+ Args:
+ task: The SweTask to evaluate.
+
+ Returns:
+ The recording provider, so tests can inspect captured execs and uploads.
+ """
+ provider = _RecordingProvider(report=_report("pkg/test_x.py::a", "pkg/test_x.py::b"))
+ harness = NVInternalHarness()
+ env = await AsyncSweEnvironment.start(provider, harness.build_spec(task))
+ try:
+ await harness.reset_repo(env, task)
+ await harness.materialize(env, task)
+ await harness.run_eval(env, task)
+ finally:
+ await env.cleanup()
+ return provider
+
+
+def test_materialize_reads_dotted_script_keys():
+ """``materialize`` uploads scripts stored under the dotted keys ``run_script.sh`` / ``parsing_script.py``."""
+ task = _task(
+ repo_workdir="/app",
+ metadata={
+ "run_script.sh": "echo DOTTED_RUN\n",
+ "parsing_script.py": "print('DOTTED_PARSE')\n",
+ "selected_test_files_to_run": ["pkg/test_x.py"],
+ },
+ )
+ provider = asyncio.run(_run_recording(task))
+ assert provider.uploads["/root/run_script.sh"] == "echo DOTTED_RUN\n"
+ assert provider.uploads["/root/parsing_script.py"] == "print('DOTTED_PARSE')\n"
+
+
+def test_materialize_dotted_keys_take_precedence_over_extensionless():
+ """When both dotted and extensionless script keys are present, the dotted keys win."""
+ task = _task(
+ repo_workdir="/app",
+ metadata={
+ "run_script.sh": "echo DOTTED\n",
+ "run_script": "echo EXTLESS\n",
+ "parsing_script.py": "print('DOTTED')\n",
+ "parsing_script": "print('EXTLESS')\n",
+ "selected_test_files_to_run": ["pkg/test_x.py"],
+ },
+ )
+ provider = asyncio.run(_run_recording(task))
+ assert provider.uploads["/root/run_script.sh"] == "echo DOTTED\n"
+ assert provider.uploads["/root/parsing_script.py"] == "print('DOTTED')\n"
+
+
+def test_materialize_falls_back_to_extensionless_keys():
+ """When only the extensionless script keys are present, they are used."""
+ task = _task(
+ repo_workdir="/app",
+ metadata={
+ "run_script": "echo EXTLESS_RUN\n",
+ "parsing_script": "print('EXTLESS_PARSE')\n",
+ "selected_test_files_to_run": ["pkg/test_x.py"],
+ },
+ )
+ provider = asyncio.run(_run_recording(task))
+ assert provider.uploads["/root/run_script.sh"] == "echo EXTLESS_RUN\n"
+ assert provider.uploads["/root/parsing_script.py"] == "print('EXTLESS_PARSE')\n"
+
+
+# ---- hops run in /app -------------------------------------------------------
+
+
+def test_nv_workdir_defaults_to_app():
+ """``_nv_workdir`` maps the generic /testbed default (or empty) to /app, honoring pinned paths."""
+ assert _nv_workdir(_task(repo_workdir="/testbed")) == NV_DEFAULT_WORKDIR
+ assert _nv_workdir(_task(repo_workdir="")) == NV_DEFAULT_WORKDIR
+ # A row that pins a non-default workdir is honored.
+ assert _nv_workdir(_task(repo_workdir="/srv/repo")) == "/srv/repo"
+ assert _nv_workdir(_task(repo_workdir="/app")) == "/app"
+
+
+def test_build_spec_workdir_defaults_to_app_for_generic_default():
+ """``build_spec`` rewrites the generic /testbed default workdir to /app."""
+ spec = NVInternalHarness().build_spec(_task(repo_workdir="/testbed"))
+ assert spec.workdir == NV_DEFAULT_WORKDIR
+
+
+def test_all_hops_run_in_app_for_generic_default():
+ """With the generic /testbed default, every reset/apply/run/parse/cat hop runs in /app."""
+ task = _task(
+ repo_workdir="/testbed",
+ metadata={
+ "run_script.sh": "echo run\n",
+ "parsing_script.py": "import sys\n",
+ "selected_test_files_to_run": ["pkg/test_x.py"],
+ },
+ )
+ provider = asyncio.run(_run_recording(task))
+ cwds = {cwd for _, cwd in provider.execs}
+ assert cwds == {NV_DEFAULT_WORKDIR}
+ # Spot-check that the key hops were exercised in /app.
+ by_cwd = {cmd: cwd for cmd, cwd in provider.execs}
+ assert any("git reset --hard" in cmd and cwd == "/app" for cmd, cwd in provider.execs)
+ assert any("git apply" in cmd and cwd == "/app" for cmd, cwd in provider.execs)
+ assert any("run_script.sh" in cmd and cwd == "/app" for cmd, cwd in provider.execs)
+ assert any("parsing_script.py" in cmd and cwd == "/app" for cmd, cwd in provider.execs)
+ assert by_cwd["cat /root/output.json"] == "/app"
+
+
+def test_all_hops_honor_explicit_non_default_workdir():
+ """A row that pins ``repo_workdir`` to a non-default path runs every hop there."""
+ task = _task(
+ repo_workdir="/srv/repo",
+ metadata={
+ "run_script.sh": "echo run\n",
+ "parsing_script.py": "import sys\n",
+ "selected_test_files_to_run": ["pkg/test_x.py"],
+ },
+ )
+ provider = asyncio.run(_run_recording(task))
+ assert {cwd for _, cwd in provider.execs} == {"/srv/repo"}
diff --git a/responses_api_agents/swe_env/tests/test_r2egym.py b/responses_api_agents/swe_env/tests/test_r2egym.py
new file mode 100644
index 0000000000..52b33348b3
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/test_r2egym.py
@@ -0,0 +1,390 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests for the r2e-gym nested harness, driven by a FakeSandbox provider.
+
+r2e-gym is a nested-harness family. These tests cover spec construction, the
+apptainer-only provider gate, the agent-phase test-hiding command shape, and
+report parsing fed a scripted ``report.json``.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+from pathlib import Path
+
+from nemo_gym.sandbox import (
+ SandboxExecResult,
+ SandboxHandle,
+ SandboxStatus,
+ register_provider,
+)
+from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+from responses_api_agents.swe_env.grading import reward_from_report
+from responses_api_agents.swe_env.harness import EvalArtifacts, SweTask
+from responses_api_agents.swe_env.harnesses.r2egym import R2EGymHarness
+
+
+class _FakeProvider:
+ """Scripted provider: the eval command returns a canned rc; ``cat`` returns the report.
+
+ Records the commands it executes and the files uploaded so tests can assert
+ on the in-sandbox side effects (e.g. that ``materialize`` writes the
+ predictions JSONL and that ``reset_repo`` issues no ``git reset``).
+ """
+
+ name = "fake-r2egym"
+
+ def __init__(self, *, report_text="", eval_rc=0, **_):
+ """Initialize the scripted provider.
+
+ Args:
+ report_text: Text returned by any ``cat`` command (the report).
+ eval_rc: Return code for the ``run_local_evaluation.py`` command.
+ """
+ self._report_text = report_text
+ self._eval_rc = eval_rc
+ self.commands: list[str] = []
+ self.uploads: dict[str, str] = {}
+
+ async def create(self, spec):
+ return SandboxHandle(sandbox_id="fake", provider_name=self.name, raw={"workdir": spec.workdir})
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ self.commands.append(command)
+ if command.startswith("cat "):
+ return SandboxExecResult(stdout=self._report_text, stderr="", return_code=0)
+ if "run_local_evaluation.py" in command:
+ return SandboxExecResult(stdout="eval done", stderr="", return_code=self._eval_rc)
+ return SandboxExecResult(stdout="", stderr="", return_code=0)
+
+ async def upload_file(self, handle, source_path, target_path):
+ self.uploads[target_path] = Path(source_path).read_text()
+ return None
+
+ async def download_file(self, *a, **k):
+ return None
+
+ async def status(self, handle):
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ return None
+
+ async def aclose(self):
+ return None
+
+
+register_provider("fake-r2egym", _FakeProvider, override=True)
+
+
+def _task(**overrides) -> SweTask:
+ """Build an r2e-gym ``SweTask`` with sensible defaults.
+
+ Args:
+ **overrides: Field values overriding the defaults.
+
+ Returns:
+ SweTask: A task populated from the defaults merged with overrides.
+ """
+ base = dict(
+ instance_id="r2e__pkg-42",
+ image="img:tag",
+ base_commit="abc123",
+ repo_workdir="/testbed",
+ model_patch="diff --git a/x b/x\n",
+ fail_to_pass=["t::a"],
+ pass_to_pass=["t::b"],
+ benchmark="r2e-gym",
+ )
+ base.update(overrides)
+ return SweTask(**base)
+
+
+def _report(instance_id: str, resolved: bool) -> str:
+ """Build a serialized nested-harness ``report.json`` for one instance.
+
+ Args:
+ instance_id: The instance id keying the report entry.
+ resolved: Whether the instance is marked resolved.
+
+ Returns:
+ str: The JSON-encoded report.
+ """
+ return json.dumps(
+ {
+ instance_id: {
+ "resolved": resolved,
+ "tests_status": {"FAIL_TO_PASS": {"success": ["t::a"], "failure": []}},
+ }
+ }
+ )
+
+
+# ---- spec + provider gate ---------------------------------------------------
+
+
+def test_harness_identity():
+ harness = R2EGymHarness()
+ assert harness.name == "r2e-gym"
+ assert harness.grade_strategy == "nested-harness"
+
+
+def test_build_spec_mounts_setup_dir():
+ harness = R2EGymHarness()
+ spec = harness.build_spec(_task(metadata={"r2egym_setup_dir": "/abs/setup"}))
+ assert spec.image == "img:tag"
+ assert spec.workdir == "/testbed"
+ assert spec.metadata["instance_id"] == "r2e__pkg-42"
+ assert spec.metadata["harness"] == "r2e-gym"
+ mounts = spec.provider_options["mounts"]
+ # Bind-mounted at both /r2egym_setup and its original absolute path.
+ assert {"src": "/abs/setup", "dst": "/r2egym_setup"} in mounts
+ assert {"src": "/abs/setup", "dst": "/abs/setup"} in mounts
+
+
+def test_build_spec_truncates_long_instance_id():
+ harness = R2EGymHarness()
+ spec = harness.build_spec(_task(instance_id="x" * 100))
+ assert len(spec.metadata["instance_id"]) == 63
+
+
+def test_supports_provider_apptainer_only():
+ harness = R2EGymHarness()
+ assert harness.supports_provider("apptainer") is True
+ assert harness.supports_provider("docker") is False
+ assert harness.supports_provider("fake-r2egym") is False
+
+
+def test_hide_eval_tests_commands_shape():
+ harness = R2EGymHarness()
+ commands = harness.hide_eval_tests_commands()
+ # One command per checkout root (root, /root, /testbed).
+ assert len(commands) == 3
+ joined = " ".join(commands)
+ assert "rm -rf /r2e_tests" in joined
+ assert "rm -rf /root/r2e_tests" in joined
+ assert "rm -rf /testbed/r2e_tests" in joined
+ # Substring guard before deleting run_tests.sh.
+ assert "grep -qs r2e_tests" in commands[0]
+
+
+# ---- grade() over the nested report.json ------------------------------------
+
+
+def test_grade_resolved_from_report():
+ harness = R2EGymHarness()
+ report = _report("r2e__pkg-42", resolved=True)
+ out = harness.grade(_task(), EvalArtifacts(test_output=report, return_code=0, raw={"report_json": report}))
+ assert out.resolved is True
+ assert out.patch_exists is True
+ assert reward_from_report(out) == 1.0
+
+
+def test_grade_unresolved_from_report():
+ harness = R2EGymHarness()
+ report = _report("r2e__pkg-42", resolved=False)
+ out = harness.grade(_task(), EvalArtifacts(test_output=report, return_code=0, raw={"report_json": report}))
+ assert out.resolved is False
+ assert reward_from_report(out) == 0.0
+
+
+def test_grade_single_entry_fallback_on_key_mismatch():
+ harness = R2EGymHarness()
+ # Report keyed by a different id than the task; sole entry is used.
+ report = _report("some-other-id", resolved=True)
+ out = harness.grade(_task(), EvalArtifacts(test_output=report, return_code=0, raw={"report_json": report}))
+ assert out.resolved is True
+
+
+def test_grade_masks_on_infra_error():
+ harness = R2EGymHarness()
+ out = harness.grade(_task(), EvalArtifacts(test_output="", return_code=1, raw={"error_type": "timeout"}))
+ assert out.error_kind == "timeout"
+ assert reward_from_report(out) == 0.0
+
+
+def test_grade_unparseable_report_is_eval_error():
+ harness = R2EGymHarness()
+ out = harness.grade(_task(), EvalArtifacts(test_output="not json", return_code=0, raw={"report_json": "not json"}))
+ assert out.error_kind == "eval_error"
+ assert reward_from_report(out) == 0.0
+
+
+# ---- run_eval over the FakeSandbox ------------------------------------------
+
+
+def _run_eval(report_text: str, eval_rc: int = 0) -> EvalArtifacts:
+ """Run the harness eval over the FakeSandbox and return its artifacts.
+
+ Args:
+ report_text: The report contents the provider returns for ``cat``.
+ eval_rc: Return code for the nested eval command.
+
+ Returns:
+ EvalArtifacts: The artifacts produced by ``run_eval``.
+ """
+
+ async def _go() -> EvalArtifacts:
+ harness = R2EGymHarness()
+ task = _task()
+ provider = {"fake-r2egym": {"report_text": report_text, "eval_rc": eval_rc}}
+ env = await AsyncSweEnvironment.start(provider, harness.build_spec(task))
+ try:
+ return await harness.run_eval(env, task)
+ finally:
+ await env.cleanup()
+
+ return asyncio.run(_go())
+
+
+def test_run_eval_then_grade_resolved():
+ report = _report("r2e__pkg-42", resolved=True)
+ artifacts = _run_eval(report)
+ assert artifacts.return_code == 0
+ assert artifacts.patch_applied is True
+ out = R2EGymHarness().grade(_task(), artifacts)
+ assert out.resolved is True
+
+
+def test_run_eval_eval_failure_marks_not_applied():
+ artifacts = _run_eval("", eval_rc=1)
+ assert artifacts.return_code == 1
+ assert artifacts.patch_applied is False
+
+
+# ---- materialize writes the predictions JSONL ------------------------------
+
+
+def test_materialize_writes_predictions_jsonl():
+ # The model patch is delivered to the nested grader via a SWE-bench
+ # predictions JSONL keyed by instance_id (--predictions_path), not a bare
+ # /root/patch.diff.
+ async def _go() -> dict[str, str]:
+ harness = R2EGymHarness()
+ task = _task(model_patch="diff --git a/x b/x\n+new line\n")
+ env = await AsyncSweEnvironment.start({"fake-r2egym": {}}, harness.build_spec(task))
+ try:
+ await harness.materialize(env, task)
+ return dict(env.sandbox._provider.uploads)
+ finally:
+ await env.cleanup()
+
+ uploads = asyncio.run(_go())
+ assert "/root/predictions.jsonl" in uploads
+ record = json.loads(uploads["/root/predictions.jsonl"])
+ assert record["instance_id"] == "r2e__pkg-42"
+ assert record["model_patch"] == "diff --git a/x b/x\n+new line\n"
+ assert record["model_name_or_path"] == "nemo-gym"
+ # The bare patch.diff path is NOT written for r2e-gym.
+ assert "/root/patch.diff" not in uploads
+
+
+def _r2e_prediction(model_patch: str) -> dict:
+ """Materialize a task with the given patch and return the prediction record.
+
+ Args:
+ model_patch: The model patch placed on the task.
+
+ Returns:
+ dict: The prediction record decoded from the uploaded JSONL.
+ """
+
+ async def _go() -> str:
+ harness = R2EGymHarness()
+ task = _task(model_patch=model_patch)
+ env = await AsyncSweEnvironment.start({"fake-r2egym": {}}, harness.build_spec(task))
+ try:
+ await harness.materialize(env, task)
+ return env.sandbox._provider.uploads["/root/predictions.jsonl"]
+ finally:
+ await env.cleanup()
+
+ return json.loads(asyncio.run(_go()))
+
+
+def test_materialize_normalizes_patch_trailing_newline():
+ # A non-empty patch missing its trailing newline gets one appended before
+ # being handed to the nested grader, so the upstream ``git apply`` does not
+ # fail.
+ assert _r2e_prediction("diff --git a/x b/x")["model_patch"] == "diff --git a/x b/x\n"
+
+
+def test_materialize_empty_patch_stays_empty():
+ # An empty patch stays "" (only a truthy patch is normalized) — it must not
+ # become a bare "\n".
+ assert _r2e_prediction("")["model_patch"] == ""
+
+
+def test_materialize_predictions_path_feeds_run_eval():
+ # The path materialize writes must match the --predictions_path run_eval
+ # passes to run_local_evaluation.py, or the patch is never read.
+ harness = R2EGymHarness()
+
+ async def _go() -> list[str]:
+ task = _task()
+ provider = {"fake-r2egym": {"report_text": _report("r2e__pkg-42", True)}}
+ env = await AsyncSweEnvironment.start(provider, harness.build_spec(task))
+ try:
+ await harness.materialize(env, task)
+ await harness.run_eval(env, task)
+ return list(env.sandbox._provider.commands)
+ finally:
+ await env.cleanup()
+
+ commands = asyncio.run(_go())
+ eval_cmd = next(c for c in commands if "run_local_evaluation.py" in c)
+ assert "--predictions_path /root/predictions.jsonl" in eval_cmd
+
+
+# ---- reset_repo is a no-op for r2e-gym --------------------------------------
+
+
+def test_reset_repo_is_noop():
+ # No host-orchestrated reset happens: the nested run_local_evaluation resets
+ # inside its own container. The base `git reset --hard ` must NOT
+ # fire for r2e-gym.
+ async def _go() -> list[str]:
+ harness = R2EGymHarness()
+ task = _task(base_commit="deadbeef")
+ env = await AsyncSweEnvironment.start({"fake-r2egym": {}}, harness.build_spec(task))
+ try:
+ await harness.reset_repo(env, task)
+ return list(env.sandbox._provider.commands)
+ finally:
+ await env.cleanup()
+
+ commands = asyncio.run(_go())
+ assert all("git reset" not in c for c in commands)
+
+
+# ---- setup-dir mount is on the channel the apptainer provider consumes ------
+
+
+def test_build_spec_mount_consumed_by_apptainer_provider(tmp_path):
+ # The provider reads provider_options["mounts"]; assert the venv setup dir is
+ # bound in via the SAME channel _mount_binds reads, so {setup}/R2E-Gym/venv
+ # is actually available in the container (no dead no-op). Use a real host dir
+ # so the canonical self-bind survives the dataset self-bind guard (which only
+ # drops self-binds whose source does not exist on the host).
+ from responses_api_agents.swe_env.providers.apptainer_provider import ApptainerSandboxProvider
+
+ setup = tmp_path / "setup"
+ setup.mkdir()
+ harness = R2EGymHarness()
+ spec = harness.build_spec(_task(metadata={"r2egym_setup_dir": str(setup)}))
+ binds = ApptainerSandboxProvider._mount_binds(spec)
+ assert f"{setup}:/r2egym_setup" in binds
+ assert f"{setup}:{setup}" in binds
diff --git a/responses_api_agents/swe_env/tests/test_swe_bench_ext.py b/responses_api_agents/swe_env/tests/test_swe_bench_ext.py
new file mode 100644
index 0000000000..b81e1810a2
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/test_swe_bench_ext.py
@@ -0,0 +1,402 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Tests for the swe-bench-ext harness grading.
+
+These cover two grading behaviors:
+
+* ``grade`` delegates to the vendored lighthouse parser
+ (``parse_and_check_tests``) — so junit-xml parsing, ``normalize_test_id`` plus
+ 4-stage fuzzy matching, the 20+ framework dispatch, and the
+ ``::build``/``::compile`` synthetic-PASS injection all drive ``resolved``.
+ Recorded fixture logs (one per parser path) anchor the expectation.
+* ``resolved`` is the parser's verdict only; a failed ``git apply`` is recorded
+ in ``patch_applied`` but never gates ``resolved``.
+
+The harness is flat / host-graded (no nested container), so ``run_eval`` runs
+against a scripted ``FakeSandbox`` rather than a real image.
+"""
+
+from __future__ import annotations
+
+import asyncio
+from pathlib import Path
+
+from nemo_gym.sandbox import (
+ SandboxExecResult,
+ SandboxHandle,
+ SandboxStatus,
+ register_provider,
+)
+from responses_api_agents.swe_env.grading import reward_from_report
+from responses_api_agents.swe_env.harness import EvalArtifacts, SweTask
+from responses_api_agents.swe_env.harnesses.swe_bench_ext import SweBenchExtHarness
+
+
+_FIXTURES = Path(__file__).parent / "fixtures" / "swe_bench_ext"
+
+
+def _fixture(name: str) -> str:
+ """Read a recorded fixture log by file name.
+
+ Args:
+ name: The fixture file name under the ``swe_bench_ext`` fixtures dir.
+
+ Returns:
+ str: The fixture file contents.
+ """
+ return (_FIXTURES / name).read_text()
+
+
+def _task(**overrides) -> SweTask:
+ """Build a swe-bench-ext ``SweTask`` with sensible defaults.
+
+ Args:
+ **overrides: Field values overriding the defaults.
+
+ Returns:
+ SweTask: A task populated from the defaults merged with overrides.
+ """
+ base = dict(
+ instance_id="repo__inst-1",
+ image="img:tag",
+ base_commit="abc123",
+ repo_workdir="/testbed",
+ test_command="python -m pytest -rA -q",
+ test_framework="pytest",
+ model_patch="diff --git a/x b/x\n",
+ fail_to_pass=["tests/test_core.py::test_fix_applied"],
+ pass_to_pass=["tests/test_core.py::test_regression_guard"],
+ benchmark="swe-bench-ext",
+ )
+ base.update(overrides)
+ return SweTask(**base)
+
+
+def _artifacts(test_output: str, *, patch_applied: bool = True, error_type=None) -> EvalArtifacts:
+ """Build ``EvalArtifacts`` for a graded run.
+
+ Args:
+ test_output: The captured test transcript handed to the parser.
+ patch_applied: Whether the model patch applied cleanly.
+ error_type: Infrastructure error kind, or None for a clean run.
+
+ Returns:
+ EvalArtifacts: The artifacts passed to ``grade``.
+ """
+ return EvalArtifacts(
+ test_output=test_output,
+ return_code=0,
+ patch_applied=patch_applied,
+ raw={"error_type": error_type},
+ )
+
+
+# --- vendored parser drives resolved ----------------------------------------
+
+
+def test_grade_junit_xml_resolved():
+ """junit-xml parsing + fuzzy id matching resolves a clean F2P/P2P pass."""
+ harness = SweBenchExtHarness()
+ report = harness.grade(_task(), _artifacts(_fixture("pytest_junit.xml")))
+ assert report.resolved is True
+ assert reward_from_report(report) == 1.0
+ # The parser report is surfaced for inspection.
+ assert report.tests_status["framework"] == "pytest"
+ assert report.tests_status["f2p_passed"] == 1
+ assert report.tests_status["p2p_passed"] == 1
+
+
+def test_grade_junit_xml_unresolved_when_p2p_fails():
+ harness = SweBenchExtHarness()
+ task = _task(pass_to_pass=["tests/test_core.py::test_unrelated_broken"])
+ report = harness.grade(task, _artifacts(_fixture("pytest_junit.xml")))
+ assert report.resolved is False
+ assert reward_from_report(report) == 0.0
+
+
+def test_grade_pytest_text_fuzzy_id_match():
+ """Normalized/fuzzy id matching: ``src/pkg/...py::test`` log id resolves a
+ differently-delimited expected id via normalize_test_id."""
+ harness = SweBenchExtHarness()
+ task = _task(
+ fail_to_pass=["src/pkg/tests/test_widget.py::test_alpha"],
+ pass_to_pass=["src/pkg/tests/test_widget.py::test_beta"],
+ )
+ report = harness.grade(task, _artifacts(_fixture("pytest_text_fuzzy.txt")))
+ assert report.resolved is True
+ assert reward_from_report(report) == 1.0
+
+
+def test_grade_pytest_text_unresolved_when_f2p_fails():
+ harness = SweBenchExtHarness()
+ task = _task(
+ fail_to_pass=["src/pkg/tests/test_widget.py::test_gamma"],
+ pass_to_pass=["src/pkg/tests/test_widget.py::test_beta"],
+ )
+ report = harness.grade(task, _artifacts(_fixture("pytest_text_fuzzy.txt")))
+ assert report.resolved is False
+
+
+def test_grade_build_synthetic_pass_injection():
+ """An F2P entry ending ``::build`` not present in the parsed output is
+ injected as PASSED (synthetic build/compile handling)."""
+ harness = SweBenchExtHarness()
+ task = _task(
+ fail_to_pass=["src/pkg/tests/test_widget.py::test_alpha", "mypkg::build"],
+ pass_to_pass=["src/pkg/tests/test_widget.py::test_beta"],
+ )
+ report = harness.grade(task, _artifacts(_fixture("pytest_text_fuzzy.txt")))
+ assert report.resolved is True
+ assert report.tests_status["fail_to_pass_results"]["mypkg::build"] == "PASSED"
+
+
+def test_grade_non_pytest_framework_go_json():
+ """A non-pytest framework (``go``) dispatches to the go-json parser."""
+ harness = SweBenchExtHarness()
+ task = _task(
+ test_framework="go",
+ fail_to_pass=["github.com/acme/widget::TestAlpha"],
+ pass_to_pass=["github.com/acme/widget::TestBeta"],
+ )
+ report = harness.grade(task, _artifacts(_fixture("go_json.txt")))
+ assert report.resolved is True
+ assert report.tests_status["framework"] == "go"
+
+
+def test_grade_non_pytest_framework_go_json_unresolved():
+ harness = SweBenchExtHarness()
+ task = _task(
+ test_framework="go",
+ fail_to_pass=["github.com/acme/widget::TestGamma"],
+ pass_to_pass=["github.com/acme/widget::TestBeta"],
+ )
+ report = harness.grade(task, _artifacts(_fixture("go_json.txt")))
+ assert report.resolved is False
+
+
+# --- empty framework is passed VERBATIM (NOT coerced to pytest) --------------
+
+
+def test_grade_empty_framework_passed_verbatim_not_coerced_to_pytest():
+ """``test_framework`` is passed through UNCHANGED — an empty framework reaches
+ ``parse_and_check_tests`` as ``""`` and hits the parser's auto-detect path, NOT
+ the pytest junit-xml parser.
+
+ Coercing ``""`` -> ``"pytest"`` would let junit-xml parse and report
+ ``resolved`` for an instance that should auto-detect. We assert the framework
+ reaches the parser verbatim (recorded in ``report.framework``) and that
+ junit-xml is therefore NOT parsed under an empty framework.
+ """
+ harness = SweBenchExtHarness()
+ task = _task(test_framework="")
+ report = harness.grade(task, _artifacts(_fixture("pytest_junit.xml")))
+ # Framework recorded verbatim — not silently rewritten to "pytest".
+ assert report.tests_status["framework"] == ""
+ # Auto-detect path does not understand junit-xml -> nothing parsed -> unresolved.
+ assert report.tests_status["parsed_count"] == 0
+ assert report.resolved is False
+
+
+def test_grade_empty_framework_uses_autodetect_path():
+ """An empty framework grades via parse_test_output's auto-detect path (TAP /
+ Mocha-Hardhat console) when the instance ships no framework. Here a TAP
+ transcript resolves without any framework hint."""
+ harness = SweBenchExtHarness()
+ tap_output = (
+ "<<>>\n"
+ "TAP version 13\n"
+ "1..2\n"
+ "ok 1 - test_fix_applied\n"
+ "ok 2 - test_regression_guard\n"
+ "<<>>\n"
+ )
+ task = _task(
+ test_framework="",
+ fail_to_pass=["test_fix_applied"],
+ pass_to_pass=["test_regression_guard"],
+ )
+ report = harness.grade(task, _artifacts(tap_output))
+ assert report.tests_status["framework"] == ""
+ assert report.tests_status["parsed_count"] >= 2
+ assert report.resolved is True
+
+
+def test_run_eval_and_grade_share_framework_value():
+ """run_eval (flag/result-file selection) and grade (parsing) use the SAME
+ framework. With an empty framework, run_eval must NOT inject pytest's
+ ``--junitxml`` flag and must wrap the bare command, and grade must parse under
+ ``""`` — proving the two share ``_resolve_framework`` rather than diverging on a
+ pytest default."""
+ task = _task(test_framework="", test_command="run-my-tests")
+ _, _, provider = _run_eval(task, test_output="", run_cmd="run-my-tests")
+ eval_cmds = [c for c in provider.commands if "run-my-tests" in c]
+ assert eval_cmds, "expected the bare framework command to be wrapped"
+ wrapped = eval_cmds[-1]
+ # Empty framework => default framework config => no output flag, no result file.
+ assert "--junitxml" not in wrapped
+ assert "<<>>" not in wrapped
+ # The mkdir parent-dir creation is present regardless.
+ assert "mkdir -p /workspace/test-results" in wrapped
+
+
+# --- patch_applied does not gate resolved -----------------------------------
+
+
+def test_grade_resolved_even_when_patch_apply_failed():
+ """Grading is on tests ONLY; a failed apply is recorded but never flips a
+ tests-passing run to unresolved."""
+ harness = SweBenchExtHarness()
+ report = harness.grade(_task(), _artifacts(_fixture("pytest_junit.xml"), patch_applied=False))
+ assert report.patch_applied is False
+ assert report.resolved is True
+ assert reward_from_report(report) == 1.0
+
+
+# --- infra masking (unchanged behavior) -------------------------------------
+
+
+def test_grade_masks_on_infra_error():
+ harness = SweBenchExtHarness()
+ report = harness.grade(_task(), _artifacts("", error_type="timeout"))
+ assert report.error_kind == "timeout"
+ assert reward_from_report(report) == 0.0
+
+
+# --- run_eval against a scripted FakeSandbox --------------------------------
+
+
+class _FakeExtProvider:
+ """Scripted provider that records git-apply attempts and returns a transcript.
+
+ Args:
+ test_output: The transcript returned for the wrapped eval command.
+ apply_rc: Return code for ``git apply`` commands.
+ run_cmd: Substring identifying the wrapped eval command.
+ """
+
+ name = "fake-ext"
+
+ def __init__(self, *, test_output="", apply_rc=0, run_cmd="pytest", **_):
+ self._test_output = test_output
+ self._apply_rc = apply_rc
+ # Marker that identifies the wrapped eval command (defaults to the pytest
+ # command); tests with a custom command pass run_cmd.
+ self._run_cmd = run_cmd
+ self.commands: list[str] = []
+ self.uploaded: dict[str, str] = {}
+
+ async def create(self, spec):
+ return SandboxHandle(sandbox_id="fake", provider_name=self.name, raw={"workdir": spec.workdir})
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ self.commands.append(command)
+ if "git apply" in command:
+ return SandboxExecResult(stdout="", stderr="", return_code=self._apply_rc)
+ if self._run_cmd in command:
+ return SandboxExecResult(stdout=self._test_output, stderr="", return_code=0)
+ return SandboxExecResult(stdout="", stderr="", return_code=0)
+
+ async def upload_file(self, handle, local_path, remote_path):
+ try:
+ with open(local_path, encoding="utf-8") as fh:
+ self.uploaded[remote_path] = fh.read()
+ except OSError:
+ self.uploaded[remote_path] = ""
+ return None
+
+ async def download_file(self, *a, **k):
+ return None
+
+ async def status(self, handle):
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ return None
+
+ async def aclose(self):
+ return None
+
+
+register_provider("fake-ext", _FakeExtProvider, override=True)
+
+
+def _run_eval(task: SweTask, *, test_output: str, apply_rc: int = 0, run_cmd: str = "pytest"):
+ """Run the harness through a scripted provider and return the run outputs.
+
+ Args:
+ task: The task to evaluate.
+ test_output: The transcript the provider returns for the eval command.
+ apply_rc: Return code for ``git apply`` commands.
+ run_cmd: Substring identifying the wrapped eval command.
+
+ Returns:
+ tuple: The harness, the produced ``EvalArtifacts``, and the provider
+ instance (for command inspection).
+ """
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ async def run():
+ harness = SweBenchExtHarness()
+ env = await AsyncSweEnvironment.start(
+ {"fake-ext": {"test_output": test_output, "apply_rc": apply_rc, "run_cmd": run_cmd}},
+ harness.build_spec(task),
+ )
+ await harness.materialize(env, task)
+ artifacts = await harness.run_eval(env, task)
+ return harness, artifacts, env.sandbox._provider
+
+ return asyncio.run(run())
+
+
+def test_run_eval_uses_legacy_apply_flags_and_grades_resolved():
+ task = _task()
+ harness, artifacts, provider = _run_eval(task, test_output=_fixture("pytest_junit.xml"))
+ apply_cmds = [c for c in provider.commands if "git apply" in c]
+ assert apply_cmds, "expected a git-apply attempt"
+ # The git-apply flag set, with no --3way fallback.
+ assert all("--reject --recount --ignore-space-change --ignore-whitespace" in c for c in apply_cmds)
+ assert all("--3way" not in c for c in apply_cmds)
+ assert artifacts.patch_applied is True
+ report = harness.grade(task, artifacts)
+ assert report.resolved is True
+
+
+def test_run_eval_apply_failure_still_resolves_on_tests():
+ # End-to-end through run_eval -> grade: a failed apply records
+ # patch_applied=False but a tests-passing run still resolves.
+ task = _task()
+ harness, artifacts, _ = _run_eval(task, test_output=_fixture("pytest_junit.xml"), apply_rc=1)
+ assert artifacts.patch_applied is False
+ report = harness.grade(task, artifacts)
+ assert report.patch_applied is False
+ assert report.resolved is True
+
+
+def test_run_eval_wraps_command_with_structured_output_and_markers():
+ # run_eval wraps the command — add the structured-output flag (--junitxml) via
+ # get_test_command_with_output and run between the SWE_BENCH_EXT markers (plus
+ # result-file dump), so parse_and_check_tests receives junit-xml / marked
+ # output rather than raw "-rA" text it cannot parse.
+ task = _task()
+ _, _, provider = _run_eval(task, test_output=_fixture("pytest_junit.xml"))
+ eval_cmds = [c for c in provider.commands if "pytest" in c and "git apply" not in c]
+ assert eval_cmds, "expected a wrapped pytest eval command"
+ wrapped = eval_cmds[-1]
+ assert "<<>>" in wrapped
+ assert "<<>>" in wrapped
+ assert "--junitxml=" in wrapped # structured-output flag from get_test_command_with_output
+ assert "<<>>" in wrapped # junit result-file dumped for the parser
+ # The result-file parent dir is created first.
+ assert "mkdir -p /workspace/test-results" in wrapped
diff --git a/responses_api_agents/swe_env/tests/test_swe_env.py b/responses_api_agents/swe_env/tests/test_swe_env.py
new file mode 100644
index 0000000000..7f6b789c1b
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/test_swe_env.py
@@ -0,0 +1,240 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests for the swe_env library, driven by a FakeSandbox provider."""
+
+from __future__ import annotations
+
+import asyncio
+
+import responses_api_agents.swe_env.harnesses # noqa: F401 (registers harnesses)
+from nemo_gym.sandbox import (
+ SandboxCreateError,
+ SandboxExecResult,
+ SandboxHandle,
+ SandboxStatus,
+ register_provider,
+)
+from resources_servers.swe_env.verify_task import ProviderCapabilityError, verify_task
+from responses_api_agents.swe_env import (
+ compute_resolved,
+ get_harness,
+ list_harnesses,
+ reward_from_report,
+)
+from responses_api_agents.swe_env.harness import EvalArtifacts, SweEvalReport, SweTask
+from responses_api_agents.swe_env.harnesses.swe_bench_ext import SweBenchExtHarness
+
+
+# Trailing-status pytest text (`` PASSED``) is the format the test
+# parser recognizes; node ids carry a ``.py`` path so they normalize to the
+# F2P/P2P ids below.
+_PASS_OUTPUT = "tests/test_x.py::a PASSED\ntests/test_x.py::b PASSED\n"
+_F2P_FAIL_OUTPUT = "tests/test_x.py::a FAILED\ntests/test_x.py::b PASSED\n"
+
+
+class _FakeProvider:
+ """Scripted provider: pytest commands return a canned transcript."""
+
+ name = "fake-swe"
+
+ def __init__(self, *, test_output="", test_rc=0, apply_rc=0, create_error=False, **_):
+ """Configure the scripted provider's responses.
+
+ Args:
+ test_output: Stdout returned for pytest commands.
+ test_rc: Return code returned for pytest commands.
+ apply_rc: Return code returned for ``git apply`` commands.
+ create_error: When True, ``create`` raises a SandboxCreateError.
+ **_: Ignored extra keyword arguments.
+ """
+ self._test_output = test_output
+ self._test_rc = test_rc
+ self._apply_rc = apply_rc
+ self._create_error = create_error
+
+ async def create(self, spec):
+ if self._create_error:
+ raise SandboxCreateError("simulated create failure")
+ return SandboxHandle(sandbox_id="fake", provider_name=self.name, raw={"workdir": spec.workdir})
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ if "pytest" in command:
+ return SandboxExecResult(stdout=self._test_output, stderr="", return_code=self._test_rc)
+ if "git apply" in command:
+ return SandboxExecResult(stdout="", stderr="", return_code=self._apply_rc)
+ return SandboxExecResult(stdout="", stderr="", return_code=0)
+
+ async def upload_file(self, *a, **k):
+ return None
+
+ async def download_file(self, *a, **k):
+ return None
+
+ async def status(self, handle):
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ return None
+
+ async def aclose(self):
+ return None
+
+
+register_provider("fake-swe", _FakeProvider, override=True)
+
+
+def _task(**overrides) -> SweTask:
+ """Build a SweTask with sensible defaults, overridable per keyword.
+
+ Args:
+ **overrides: Field overrides merged onto the default task fields.
+
+ Returns:
+ A SweTask configured for the swe-bench-ext benchmark.
+ """
+ base = dict(
+ instance_id="inst-1",
+ image="img:tag",
+ base_commit="abc123",
+ repo_workdir="/testbed",
+ test_command="python -m pytest -rA -q",
+ model_patch="diff --git a/x b/x\n",
+ test_framework="pytest",
+ fail_to_pass=["tests/test_x.py::a"],
+ pass_to_pass=["tests/test_x.py::b"],
+ benchmark="swe-bench-ext",
+ )
+ base.update(overrides)
+ return SweTask(**base)
+
+
+# ---- pure helpers -----------------------------------------------------------
+
+
+def test_compute_resolved():
+ """``compute_resolved`` is True only when all required tests are in the passed set."""
+ assert compute_resolved(fail_to_pass=["a"], pass_to_pass=["b"], passed=["a", "b"]) is True
+ assert compute_resolved(fail_to_pass=["a"], pass_to_pass=["b"], passed=["a"]) is False
+ assert compute_resolved(fail_to_pass=[], pass_to_pass=[], passed=["a"]) is False
+
+
+def test_reward_from_report():
+ """``reward_from_report`` is 1.0 for a resolved report and 0.0 otherwise or when masked."""
+ assert reward_from_report(SweEvalReport(instance_id="i", resolved=True)) == 1.0
+ assert reward_from_report(SweEvalReport(instance_id="i", resolved=False)) == 0.0
+ assert reward_from_report(SweEvalReport(instance_id="i", resolved=True, error_kind="sandbox")) == 0.0
+
+
+def test_registry_and_build_spec():
+ """The swe-bench-ext harness is registered and builds the expected sandbox spec."""
+ assert "swe-bench-ext" in list_harnesses()
+ harness = get_harness("swe-bench-ext")
+ assert isinstance(harness, SweBenchExtHarness)
+ spec = harness.build_spec(_task())
+ assert spec.image == "img:tag"
+ assert spec.workdir == "/testbed"
+ assert spec.metadata["instance_id"] == "inst-1"
+
+
+def test_grade_masks_on_infra_error():
+ """Grading masks an infra error to reward 0.0 and records its error kind."""
+ harness = get_harness("swe-bench-ext")
+ report = harness.grade(_task(), EvalArtifacts(test_output="", return_code=1, raw={"error_type": "timeout"}))
+ assert report.error_kind == "timeout"
+ assert reward_from_report(report) == 0.0
+
+
+# ---- verify_task orchestrator (fresh-sandbox, FakeProvider) -----------------
+
+
+def test_verify_task_resolved():
+ """``verify_task`` resolves a task whose required tests all pass."""
+ provider = {"fake-swe": {"test_output": _PASS_OUTPUT, "test_rc": 0}}
+ report = asyncio.run(verify_task(provider, _task()))
+ assert report.resolved is True
+ assert report.patch_applied is True
+ assert reward_from_report(report) == 1.0
+
+
+def test_verify_task_unresolved():
+ """``verify_task`` leaves a task unresolved when a required test fails."""
+ provider = {"fake-swe": {"test_output": _F2P_FAIL_OUTPUT, "test_rc": 1}}
+ report = asyncio.run(verify_task(provider, _task()))
+ assert report.resolved is False
+ assert reward_from_report(report) == 0.0
+
+
+def test_verify_task_empty_patch_fast_path():
+ """An empty model patch short-circuits to an unresolved report."""
+ report = asyncio.run(verify_task({"fake-swe": {}}, _task(model_patch="")))
+ assert report.patch_exists is False
+ assert report.resolved is False
+
+
+def test_verify_task_infra_error_masked():
+ """A sandbox creation failure is masked to reward 0.0 with a sandbox error kind."""
+ report = asyncio.run(verify_task({"fake-swe": {"create_error": True}}, _task()))
+ assert report.error_kind == "sandbox"
+ assert reward_from_report(report) == 0.0
+
+
+def test_verify_task_golden():
+ """Running with ``run_golden`` applies the golden patch and resolves the task."""
+ provider = {"fake-swe": {"test_output": _PASS_OUTPUT}}
+ task = _task(model_patch="", metadata={"golden_patch": "diff --git a/x b/x\n"})
+ report = asyncio.run(verify_task(provider, task, run_golden=True))
+ assert report.resolved is True
+
+
+def test_verify_task_patch_apply_failure_does_not_gate_resolved():
+ """A failed patch apply is recorded but does not gate ``resolved``.
+
+ The patch is applied best-effort and grading is based on the tests only, so a
+ failed apply (patch_applied=False) does not flip a tests-passing run to
+ unresolved.
+ """
+ provider = {"fake-swe": {"test_output": _PASS_OUTPUT, "apply_rc": 1}}
+ report = asyncio.run(verify_task(provider, _task()))
+ assert report.patch_applied is False
+ assert report.resolved is True
+ assert reward_from_report(report) == 1.0
+
+
+def test_unsupported_provider_raises():
+ """``verify_task`` raises when the harness does not support the given provider."""
+
+ class _NestedOnly(SweBenchExtHarness):
+ name = "nested-only-test"
+
+ def supports_provider(self, provider_name: str) -> bool:
+ """Report support for every provider except ``fake-swe``.
+
+ Args:
+ provider_name: The provider name being checked.
+
+ Returns:
+ True for any provider other than ``fake-swe``.
+ """
+ return provider_name != "fake-swe"
+
+ from responses_api_agents.swe_env.registry import register_harness
+
+ register_harness(_NestedOnly(), override=True)
+ task = _task(benchmark="nested-only-test")
+ try:
+ asyncio.run(verify_task({"fake-swe": {}}, task))
+ except ProviderCapabilityError:
+ return
+ raise AssertionError("expected ProviderCapabilityError")
diff --git a/responses_api_agents/swe_env/tests/test_swe_rebench.py b/responses_api_agents/swe_env/tests/test_swe_rebench.py
new file mode 100644
index 0000000000..24b86d3eb0
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/test_swe_rebench.py
@@ -0,0 +1,484 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests for the swe-rebench harness (FakeSandbox provider).
+
+A tiny fake ``agent/log_parsers.py`` is written to a tmp dir so the real
+``_load_rebench_log_parsers`` import and ``NAME_TO_PARSER`` resolution path is
+exercised end to end, then the resolved / unresolved / masked grade paths are
+driven.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import textwrap
+from pathlib import Path
+
+from nemo_gym.sandbox import (
+ SandboxExecResult,
+ SandboxHandle,
+ SandboxStatus,
+ register_provider,
+)
+from responses_api_agents.swe_env.harness import EvalArtifacts, SweTask
+from responses_api_agents.swe_env.harnesses.swe_rebench import (
+ SweRebenchHarness,
+ _normalize_test_name,
+)
+
+
+class _FakeProvider:
+ """Scripted provider: test command returns a canned transcript."""
+
+ name = "fake-rebench"
+
+ def __init__(self, *, test_output="", test_rc=0, apply_rc=0, **_):
+ """Initialize the scripted provider.
+
+ Args:
+ test_output: Transcript returned for the test command.
+ test_rc: Return code for the test command.
+ apply_rc: Return code for ``git apply`` commands.
+ """
+ self._test_output = test_output
+ self._test_rc = test_rc
+ self._apply_rc = apply_rc
+
+ async def create(self, spec):
+ raw = {"workdir": spec.workdir, "env": spec.env}
+ return SandboxHandle(sandbox_id="fake", provider_name=self.name, raw=raw)
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ if "git apply" in command:
+ return SandboxExecResult(stdout="", stderr="", return_code=self._apply_rc)
+ if "pytest" in command or "test" in command:
+ return SandboxExecResult(stdout=self._test_output, stderr="", return_code=self._test_rc)
+ return SandboxExecResult(stdout="", stderr="", return_code=0)
+
+ async def upload_file(self, *a, **k):
+ return None
+
+ async def download_file(self, *a, **k):
+ return None
+
+ async def status(self, handle):
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ return None
+
+ async def aclose(self):
+ return None
+
+
+register_provider("fake-rebench", _FakeProvider, override=True)
+
+
+class _RecordingProvider:
+ """Scripted provider that records every exec command, in order."""
+
+ name = "recording-rebench"
+ commands: list[str] = []
+ # (command, timeout_s) for every exec, so tests can assert the eval timeout
+ # is threaded into the test exec.
+ exec_calls: list[tuple[str, object]] = []
+
+ def __init__(self, *, test_output="", test_rc=0, apply_rc=0, **_):
+ """Initialize the recording provider.
+
+ Args:
+ test_output: Transcript returned for the test command.
+ test_rc: Return code for the test command.
+ apply_rc: Return code for ``git apply`` commands.
+ """
+ self._test_output = test_output
+ self._test_rc = test_rc
+ self._apply_rc = apply_rc
+
+ async def create(self, spec):
+ return SandboxHandle(sandbox_id="rec", provider_name=self.name, raw={"workdir": spec.workdir})
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ type(self).commands.append(command)
+ type(self).exec_calls.append((command, timeout_s))
+ if "git apply" in command:
+ return SandboxExecResult(stdout="", stderr="", return_code=self._apply_rc)
+ if "pytest" in command or "test" in command:
+ return SandboxExecResult(stdout=self._test_output, stderr="", return_code=self._test_rc)
+ return SandboxExecResult(stdout="", stderr="", return_code=0)
+
+ async def upload_file(self, *a, **k):
+ return None
+
+ async def download_file(self, *a, **k):
+ return None
+
+ async def status(self, handle):
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ return None
+
+ async def aclose(self):
+ return None
+
+
+register_provider("recording-rebench", _RecordingProvider, override=True)
+
+
+# A standalone log_parsers module the harness imports dynamically. The parser
+# splits " " lines into {node: STATUS} and exposes a
+# NAME_TO_PARSER registry of callables, matching the shape the harness expects.
+_FAKE_LOG_PARSERS = textwrap.dedent(
+ """
+ def parse_simple(log):
+ results = {}
+ for line in log.splitlines():
+ line = line.strip()
+ if not line:
+ continue
+ node, _, status = line.rpartition(" ")
+ if node and status:
+ results[node] = status
+ return results
+
+ NAME_TO_PARSER = {"simple": parse_simple}
+ """
+)
+
+
+def _write_fake_parsers(tmp_path: Path) -> Path:
+ """Write the fake ``agent/log_parsers.py`` module under a tmp repo dir.
+
+ Args:
+ tmp_path: The pytest tmp dir to create the repo under.
+
+ Returns:
+ Path: The created ``SWE-rebench-V2`` repo directory.
+ """
+ repo_dir = tmp_path / "SWE-rebench-V2"
+ (repo_dir / "agent").mkdir(parents=True)
+ (repo_dir / "agent" / "log_parsers.py").write_text(_FAKE_LOG_PARSERS)
+ return repo_dir
+
+
+def _task(**overrides) -> SweTask:
+ """Build a swe-rebench ``SweTask`` with sensible defaults.
+
+ Args:
+ **overrides: Field values overriding the defaults.
+
+ Returns:
+ SweTask: A task populated from the defaults merged with overrides.
+ """
+ base = dict(
+ instance_id="rebench-1",
+ image="img:tag",
+ base_commit="abc123",
+ repo_workdir="/testbed",
+ test_command="python -m pytest -rA -q",
+ model_patch="diff --git a/x b/x\n",
+ test_patch="diff --git a/t b/t\n",
+ fail_to_pass=["t::a"],
+ pass_to_pass=["t::b"],
+ benchmark="swe-rebench",
+ )
+ base.update(overrides)
+ return SweTask(**base)
+
+
+# ---- pure helpers -----------------------------------------------------------
+
+
+def test_normalize_test_name_strips_timing():
+ assert _normalize_test_name("t::a [ 12 ms ]") == "t::a"
+ assert _normalize_test_name("t::a [0.3s]") == "t::a"
+ assert _normalize_test_name("t::a in 1.2 sec") == "t::a"
+ assert _normalize_test_name("t::a (5 ms)") == "t::a"
+ assert _normalize_test_name(" t::a ") == "t::a"
+ # No timing suffix -> unchanged.
+ assert _normalize_test_name("pkg::mod::test_x") == "pkg::mod::test_x"
+
+
+def test_build_spec_sets_java_env():
+ harness = SweRebenchHarness()
+ spec = harness.build_spec(_task())
+ assert spec.env["_JAVA_OPTIONS"] == "-Djava.net.preferIPv6Addresses=false"
+ assert spec.metadata["harness"] == "swe-rebench"
+ assert spec.image == "img:tag"
+
+
+# ---- grade paths (real dynamic-import of the fake parser) --------------------
+
+
+def test_grade_resolved(tmp_path):
+ repo_dir = _write_fake_parsers(tmp_path)
+ harness = SweRebenchHarness()
+ task = _task(
+ metadata={"rebench_repo_dir": str(repo_dir), "install_config": {"log_parser": "simple"}},
+ )
+ # Both required tests pass; timing suffix on one exercises normalization.
+ artifacts = EvalArtifacts(test_output="t::a [ 12 ms ] PASSED\nt::b PASSED\n", patch_applied=True)
+ report = harness.grade(task, artifacts)
+ assert report.resolved is True
+ assert report.error_kind is None
+ assert set(report.tests_status["passed"]) == {"t::a", "t::b"}
+
+
+def test_grade_unresolved_missing_pass_to_pass(tmp_path):
+ repo_dir = _write_fake_parsers(tmp_path)
+ harness = SweRebenchHarness()
+ task = _task(
+ metadata={"rebench_repo_dir": str(repo_dir), "install_config": {"log_parser": "simple"}},
+ )
+ artifacts = EvalArtifacts(test_output="t::a PASSED\nt::b FAILED\n", patch_applied=True)
+ report = harness.grade(task, artifacts)
+ assert report.resolved is False
+ assert report.error_kind is None
+
+
+def test_grade_no_patch_applied_gate(tmp_path):
+ """``resolved`` is the test verdict ONLY and does not gate on patch_applied.
+ The report reports patch_successfully_applied=True, so even when the model
+ patch failed to apply, a run where every F2P/P2P test passes scores
+ resolved=True."""
+ repo_dir = _write_fake_parsers(tmp_path)
+ harness = SweRebenchHarness()
+ task = _task(
+ metadata={"rebench_repo_dir": str(repo_dir), "install_config": {"log_parser": "simple"}},
+ )
+ artifacts = EvalArtifacts(test_output="t::a PASSED\nt::b PASSED\n", patch_applied=False)
+ report = harness.grade(task, artifacts)
+ assert report.resolved is True
+ assert report.error_kind is None
+
+
+def test_grade_masks_missing_clone():
+ harness = SweRebenchHarness()
+ # No rebench_repo_dir in metadata -> the clone is not provisioned.
+ report = harness.grade(_task(), EvalArtifacts(test_output="t::a PASSED\n", patch_applied=True))
+ assert report.error_kind == "eval_error"
+ assert report.resolved is False
+
+
+def test_grade_masks_unknown_parser(tmp_path):
+ repo_dir = _write_fake_parsers(tmp_path)
+ harness = SweRebenchHarness()
+ task = _task(
+ metadata={"rebench_repo_dir": str(repo_dir), "install_config": {"log_parser": "does_not_exist"}},
+ )
+ report = harness.grade(task, EvalArtifacts(test_output="t::a PASSED\n", patch_applied=True))
+ assert report.error_kind == "eval_error"
+
+
+def test_grade_masks_on_infra_error():
+ harness = SweRebenchHarness()
+ report = harness.grade(_task(), EvalArtifacts(test_output="", return_code=1, raw={"error_type": "timeout"}))
+ assert report.error_kind == "timeout"
+
+
+# ---- run_eval (FakeSandbox) -------------------------------------------------
+
+
+def test_run_eval_then_grade_resolved(tmp_path):
+ repo_dir = _write_fake_parsers(tmp_path)
+ harness = SweRebenchHarness()
+ task = _task(
+ metadata={
+ "rebench_repo_dir": str(repo_dir),
+ "install_config": {"log_parser": "simple", "test_cmd": "python -m pytest -rA -q"},
+ },
+ )
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ provider = {"fake-rebench": {"test_output": "t::a PASSED\nt::b PASSED\n", "test_rc": 0}}
+
+ async def _run():
+ spec = harness.build_spec(task)
+ env = await AsyncSweEnvironment.start(provider, spec)
+ try:
+ await harness.reset_repo(env, task)
+ await harness.materialize(env, task)
+ artifacts = await harness.run_eval(env, task)
+ finally:
+ await env.cleanup()
+ return artifacts
+
+ artifacts = asyncio.run(_run())
+ assert artifacts.patch_applied is True
+ report = harness.grade(task, artifacts)
+ assert report.resolved is True
+
+
+def test_run_eval_patch_not_applied_still_grades_on_tests(tmp_path):
+ repo_dir = _write_fake_parsers(tmp_path)
+ harness = SweRebenchHarness()
+ task = _task(metadata={"rebench_repo_dir": str(repo_dir), "install_config": {"log_parser": "simple"}})
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ # apply_rc=1 -> model patch fails to apply -> patch_applied False, but grading
+ # is on the tests only (no patch_applied gate), so a run where every F2P/P2P
+ # test passes is still resolved=True.
+ provider = {"fake-rebench": {"test_output": "t::a PASSED\nt::b PASSED\n", "apply_rc": 1}}
+
+ async def _run():
+ spec = harness.build_spec(task)
+ env = await AsyncSweEnvironment.start(provider, spec)
+ try:
+ await harness.run_eval(env, task)
+ return await harness.run_eval(env, task)
+ finally:
+ await env.cleanup()
+
+ artifacts = asyncio.run(_run())
+ assert artifacts.patch_applied is False
+ assert harness.grade(task, artifacts).resolved is True
+
+
+# ---- apply order ------------------------------------------------------------
+
+
+def test_run_eval_applies_model_patch_before_test_patch(tmp_path):
+ """The model patch (/root/patch.diff) is applied BEFORE the test patch
+ (/root/test_patch.diff)."""
+ repo_dir = _write_fake_parsers(tmp_path)
+ harness = SweRebenchHarness()
+ task = _task(metadata={"rebench_repo_dir": str(repo_dir), "install_config": {"log_parser": "simple"}})
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ _RecordingProvider.commands = []
+ _RecordingProvider.exec_calls = []
+ provider = {"recording-rebench": {"test_output": "t::a PASSED\nt::b PASSED\n"}}
+
+ async def _run():
+ spec = harness.build_spec(task)
+ env = await AsyncSweEnvironment.start(provider, spec)
+ try:
+ await harness.run_eval(env, task)
+ finally:
+ await env.cleanup()
+
+ asyncio.run(_run())
+ applies = [c for c in _RecordingProvider.commands if "git apply" in c]
+ assert len(applies) == 2
+ assert "/root/patch.diff" in applies[0], applies
+ assert "/root/test_patch.diff" in applies[1], applies
+
+
+# ---- eval timeout threaded into the test exec -------------------------------
+
+
+def _rebench_test_exec_timeout(commands_and_timeouts):
+ """Return the timeout_s passed to the test exec (the one running the tests).
+
+ The test block is the only exec that is neither a ``git apply`` nor an
+ install command; in these tests the test command always contains ``pytest``.
+
+ Args:
+ commands_and_timeouts: An iterable of ``(command, timeout_s)`` pairs.
+
+ Returns:
+ The ``timeout_s`` value recorded for the test exec.
+
+ Raises:
+ AssertionError: If no test exec is found in the recorded calls.
+ """
+ for command, timeout_s in commands_and_timeouts:
+ if "git apply" not in command and ("pytest" in command or "test" in command):
+ return timeout_s
+ raise AssertionError(f"no test exec found in {commands_and_timeouts!r}")
+
+
+def test_run_eval_threads_tests_timeout_into_test_exec(tmp_path):
+ """The test exec receives timeout_s = task.metadata['tests_timeout'] when
+ present so a stuck run is bounded instead of hanging the verifier. Uses a
+ non-default value (600) so this distinguishes an explicit override from the
+ 1800 default."""
+ repo_dir = _write_fake_parsers(tmp_path)
+ harness = SweRebenchHarness()
+ task = _task(
+ metadata={
+ "rebench_repo_dir": str(repo_dir),
+ "install_config": {"log_parser": "simple", "test_cmd": "python -m pytest -rA -q"},
+ "tests_timeout": 600,
+ },
+ )
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ _RecordingProvider.commands = []
+ _RecordingProvider.exec_calls = []
+ provider = {"recording-rebench": {"test_output": "t::a PASSED\nt::b PASSED\n"}}
+
+ async def _run():
+ spec = harness.build_spec(task)
+ env = await AsyncSweEnvironment.start(provider, spec)
+ try:
+ await harness.run_eval(env, task)
+ finally:
+ await env.cleanup()
+
+ asyncio.run(_run())
+ assert _rebench_test_exec_timeout(_RecordingProvider.exec_calls) == 600
+
+
+def test_run_eval_tests_timeout_absent_defaults_to_1800(tmp_path):
+ """The timeout (default 30*60) is applied to every swe-rebench run. Rows that
+ carry no tests_timeout (including SWE-bench-Verified) still get the 1800s
+ bound rather than an unbounded (None) run."""
+ repo_dir = _write_fake_parsers(tmp_path)
+ harness = SweRebenchHarness()
+ task = _task(
+ metadata={
+ "rebench_repo_dir": str(repo_dir),
+ "install_config": {"log_parser": "simple", "test_cmd": "python -m pytest -rA -q"},
+ },
+ )
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ _RecordingProvider.commands = []
+ _RecordingProvider.exec_calls = []
+ provider = {"recording-rebench": {"test_output": "t::a PASSED\nt::b PASSED\n"}}
+
+ async def _run():
+ spec = harness.build_spec(task)
+ env = await AsyncSweEnvironment.start(provider, spec)
+ try:
+ await harness.run_eval(env, task)
+ finally:
+ await env.cleanup()
+
+ asyncio.run(_run())
+ assert _rebench_test_exec_timeout(_RecordingProvider.exec_calls) == 1800
+
+
+# ---- grading parity / empty-required ----------------------------------------
+
+
+def test_grade_empty_required_resolves_true(tmp_path):
+ """``resolved`` is purely (fail_to_pass_set <= passed) and
+ (pass_to_pass_set <= passed). With no required tests, both empty sets are
+ subsets of any passed set, so resolved=True — there is no bool(required)
+ requirement."""
+ repo_dir = _write_fake_parsers(tmp_path)
+ harness = SweRebenchHarness()
+ task = _task(
+ fail_to_pass=[],
+ pass_to_pass=[],
+ metadata={"rebench_repo_dir": str(repo_dir), "install_config": {"log_parser": "simple"}},
+ )
+ artifacts = EvalArtifacts(test_output="something PASSED\n", patch_applied=True)
+ report = harness.grade(task, artifacts)
+ assert report.resolved is True
+ assert report.error_kind is None
diff --git a/responses_api_agents/swe_env/tests/test_swebench.py b/responses_api_agents/swe_env/tests/test_swebench.py
new file mode 100644
index 0000000000..116bd9c8d9
--- /dev/null
+++ b/responses_api_agents/swe_env/tests/test_swebench.py
@@ -0,0 +1,525 @@
+# Copyright (c) 2026, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Unit tests for the nested swe-bench / swe-bench-multilingual harness.
+
+The nested families run the ``run_local_evaluation`` harness inside an apptainer
+sandbox. These tests validate provisioning (``build_spec`` / ``supports_provider``
+/ ``materialize``) and host-side ``grade`` parsing of a sample ``report.json``
+against a scripted ``FakeSandbox``.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+
+from nemo_gym.sandbox import (
+ SandboxExecResult,
+ SandboxHandle,
+ SandboxStatus,
+ register_provider,
+)
+from responses_api_agents.swe_env.grading import reward_from_report
+from responses_api_agents.swe_env.harness import EvalArtifacts, SweTask
+from responses_api_agents.swe_env.harnesses.swebench import (
+ _DATASET_PATH,
+ _PREDICTIONS_PATH,
+ _REPORT_PATH,
+ SweBenchHarness,
+)
+
+
+class _FakeProvider:
+ """Scripted sandbox provider for the nested swe-bench harness.
+
+ ``run_local_evaluation`` is a no-op, ``cat`` returns a canned report, and
+ uploaded text is recorded so ``materialize`` can be asserted.
+
+ Args:
+ report_text: Text returned by any ``cat`` command (the report contents).
+ report_rc: Return code for the ``cat`` command.
+ eval_rc: Return code for the eval/collect command.
+ """
+
+ name = "fake-swebench"
+
+ def __init__(self, *, report_text="", report_rc=0, eval_rc=0, **_):
+ self._report_text = report_text
+ self._report_rc = report_rc
+ self._eval_rc = eval_rc
+ self.uploaded: dict[str, str] = {}
+
+ async def create(self, spec):
+ return SandboxHandle(sandbox_id="fake", provider_name=self.name, raw={"workdir": spec.workdir})
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ if command.startswith("cat "):
+ return SandboxExecResult(stdout=self._report_text, stderr="", return_code=self._report_rc)
+ # The eval and collect step.
+ return SandboxExecResult(stdout="ran nested harness", stderr="", return_code=self._eval_rc)
+
+ async def upload_file(self, handle, local_path, remote_path):
+ try:
+ with open(local_path, encoding="utf-8") as fh:
+ self.uploaded[remote_path] = fh.read()
+ except OSError:
+ self.uploaded[remote_path] = ""
+ return None
+
+ async def download_file(self, *a, **k):
+ return None
+
+ async def status(self, handle):
+ return SandboxStatus.RUNNING
+
+ async def close(self, handle):
+ return None
+
+ async def aclose(self):
+ return None
+
+
+register_provider("fake-swebench", _FakeProvider, override=True)
+
+
+def _task(**overrides) -> SweTask:
+ """Build a swe-bench ``SweTask`` with sensible defaults.
+
+ Args:
+ **overrides: Field values overriding the defaults.
+
+ Returns:
+ SweTask: A task populated from the defaults merged with overrides.
+ """
+ base = dict(
+ instance_id="repo__inst-1",
+ image="img:tag",
+ base_commit="abc123",
+ repo_workdir="/testbed",
+ model_patch="diff --git a/x b/x\n",
+ fail_to_pass=["t::a"],
+ pass_to_pass=["t::b"],
+ benchmark="swe-bench",
+ split="test",
+ )
+ base.update(overrides)
+ return SweTask(**base)
+
+
+def _sample_report(instance_id: str, resolved: bool) -> str:
+ """Build a serialized nested-harness ``report.json`` for one instance.
+
+ Args:
+ instance_id: The instance id keying the report entry.
+ resolved: Whether the instance is marked resolved.
+
+ Returns:
+ str: The JSON-encoded report.
+ """
+ return json.dumps(
+ {
+ instance_id: {
+ "resolved": resolved,
+ "patch_is_None": False,
+ "patch_successfully_applied": True,
+ "tests_status": {"FAIL_TO_PASS": {"success": ["t::a"], "failure": []}},
+ }
+ }
+ )
+
+
+# ---- provisioning -----------------------------------------------------------
+
+
+def test_grade_strategy_is_nested():
+ assert SweBenchHarness("swe-bench").grade_strategy == "nested-harness"
+ assert SweBenchHarness("swe-bench-multilingual").grade_strategy == "nested-harness"
+
+
+def test_unknown_family_rejected():
+ try:
+ SweBenchHarness("not-a-family")
+ except ValueError:
+ return
+ raise AssertionError("expected ValueError for unknown family")
+
+
+def test_build_spec_image_and_mounts():
+ harness = SweBenchHarness("swe-bench")
+ task = _task(metadata={"host_setup_dir": "/host/swe_swebench_setup"})
+ spec = harness.build_spec(task)
+ assert spec.image == "img:tag"
+ assert spec.workdir == "/testbed"
+ assert spec.metadata["instance_id"] == "repo__inst-1"
+ assert spec.metadata["harness"] == "swe-bench"
+ # Mounts live on provider_options (typed dict[str, Any]) — the channel the
+ # apptainer provider consumes. metadata is dict[str, str] and never carries
+ # the mount list.
+ assert "mounts" not in spec.metadata
+ mounts = spec.provider_options["mounts"]
+ dsts = {m["dst"] for m in mounts}
+ assert "/root/dataset/data.jsonl" in dsts
+ # Host setup dir bind-mounted at both the alias and its canonical path.
+ assert "/swebench_setup" in dsts
+ assert "/host/swe_swebench_setup" in dsts
+ # Both setup-dir binds point at the host setup dir.
+ setup_binds = {m["src"] for m in mounts if m["dst"] in {"/swebench_setup", "/host/swe_swebench_setup"}}
+ assert setup_binds == {"/host/swe_swebench_setup"}
+
+
+def test_build_spec_multilingual_mount_alias():
+ harness = SweBenchHarness("swe-bench-multilingual")
+ task = _task(benchmark="swe-bench-multilingual", metadata={"host_setup_dir": "/host/ml"})
+ spec = harness.build_spec(task)
+ mounts = spec.provider_options["mounts"]
+ dsts = {m["dst"] for m in mounts}
+ # Multilingual alias plus the canonical host path.
+ assert "/swebench_multilingual_setup" in dsts
+ assert "/host/ml" in dsts
+
+
+def test_build_spec_preserves_task_provider_options():
+ # A task-supplied provider_options (e.g. instance_args) must survive; only
+ # the default mounts are filled in when absent.
+ harness = SweBenchHarness("swe-bench")
+ task = _task(
+ metadata={
+ "host_setup_dir": "/host/s",
+ "provider_options": {"instance_args": ["--nv"]},
+ }
+ )
+ spec = harness.build_spec(task)
+ assert spec.provider_options["instance_args"] == ["--nv"]
+ assert "mounts" in spec.provider_options
+
+
+def test_supports_provider_fail_fast_on_docker():
+ harness = SweBenchHarness("swe-bench")
+ assert harness.supports_provider("apptainer") is True
+ assert harness.supports_provider("docker") is False
+ assert harness.supports_provider("fake-swebench") is False
+
+
+def test_materialize_writes_predictions_jsonl():
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ provider = {"fake-swebench": {}}
+
+ async def run():
+ harness = SweBenchHarness("swe-bench")
+ task = _task()
+ env = await AsyncSweEnvironment.start(provider, harness.build_spec(task))
+ # Reach into the underlying provider instance to inspect uploads.
+ await harness.materialize(env, task)
+ return env.sandbox._provider
+
+ sandbox_provider = asyncio.run(run())
+ assert _PREDICTIONS_PATH in sandbox_provider.uploaded
+ prediction = json.loads(sandbox_provider.uploaded[_PREDICTIONS_PATH])
+ assert prediction["instance_id"] == "repo__inst-1"
+ assert prediction["model_patch"] == "diff --git a/x b/x\n"
+
+
+# ---- grade (sample report.json) ---------------------------------------------
+
+
+def test_grade_resolved_from_report():
+ harness = SweBenchHarness("swe-bench")
+ task = _task()
+ artifacts = EvalArtifacts(
+ test_output="ran",
+ return_code=0,
+ patch_applied=True,
+ raw={"error_type": None, "report_json": _sample_report(task.instance_id, True)},
+ )
+ report = harness.grade(task, artifacts)
+ assert report.resolved is True
+ assert report.patch_applied is True
+ assert report.patch_exists is True
+ assert reward_from_report(report) == 1.0
+
+
+def test_grade_unresolved_from_report():
+ harness = SweBenchHarness("swe-bench")
+ task = _task()
+ artifacts = EvalArtifacts(raw={"error_type": None, "report_json": _sample_report(task.instance_id, False)})
+ report = harness.grade(task, artifacts)
+ assert report.resolved is False
+ assert reward_from_report(report) == 0.0
+
+
+def test_grade_masks_on_infra_error():
+ harness = SweBenchHarness("swe-bench")
+ report = harness.grade(_task(), EvalArtifacts(raw={"error_type": "timeout"}))
+ assert report.error_kind == "timeout"
+ assert reward_from_report(report) == 0.0
+
+
+def test_grade_masks_on_missing_report():
+ harness = SweBenchHarness("swe-bench")
+ report = harness.grade(_task(), EvalArtifacts(raw={"error_type": None, "report_json": ""}))
+ assert report.error_kind == "eval_error"
+ assert reward_from_report(report) == 0.0
+
+
+# ---- run_eval (FakeSandbox: nested command issued, report read back) --------
+
+
+def test_run_eval_reads_report_and_grades():
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ task = _task()
+ report_text = _sample_report(task.instance_id, True)
+ provider = {"fake-swebench": {"report_text": report_text}}
+
+ async def run():
+ harness = SweBenchHarness("swe-bench")
+ env = await AsyncSweEnvironment.start(provider, harness.build_spec(task))
+ await harness.materialize(env, task)
+ artifacts = await harness.run_eval(env, task)
+ return harness.grade(task, artifacts), artifacts
+
+ report, artifacts = asyncio.run(run())
+ assert artifacts.raw["report_json"] == report_text
+ assert report.resolved is True
+ assert reward_from_report(report) == 1.0
+
+
+def test_run_eval_report_path_constant_is_stable():
+ # The collect step copies the nested harness report to this fixed path.
+ assert _REPORT_PATH == "/root/report.json"
+
+
+class _RecordingProvider(_FakeProvider):
+ """Like ``_FakeProvider`` but records every command issued to ``exec``."""
+
+ name = "rec-swebench"
+
+ def __init__(self, **kw):
+ super().__init__(**kw)
+ self.commands: list[str] = []
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ self.commands.append(command)
+ return await super().exec(handle, command, cwd=cwd, env=env, timeout_s=timeout_s, user=user)
+
+
+register_provider("rec-swebench", _RecordingProvider, override=True)
+
+
+def _eval_command(*, family="swe-bench", **task_overrides) -> str:
+ """Run run_eval through a recording provider and return the eval command string.
+
+ Args:
+ family: The swe-bench family to instantiate the harness for.
+ **task_overrides: Field values overriding the task defaults.
+
+ Returns:
+ str: The single eval command containing ``run_local_evaluation``.
+ """
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ async def run():
+ harness = SweBenchHarness(family)
+ task = _task(benchmark=family, **task_overrides)
+ env = await AsyncSweEnvironment.start({"rec-swebench": {}}, harness.build_spec(task))
+ await harness.materialize(env, task)
+ await harness.run_eval(env, task)
+ # The eval+collect step is the only non-``cat`` command issued.
+ cmds = [c for c in env.sandbox._provider.commands if "run_local_evaluation" in c]
+ return cmds[0]
+
+ return asyncio.run(run())
+
+
+def test_run_eval_command_has_uv_exports_swebench():
+ # The command exports UV_INSTALL_DIR / UV_PYTHON_INSTALL_DIR / PATH pointing
+ # at the mounted portable uv+python so the prebuilt venv resolves its
+ # hardcoded toolchain.
+ cmd = _eval_command(family="swe-bench", metadata={"setup_dir": "/swebench_setup"})
+ assert 'export UV_INSTALL_DIR="/swebench_setup/uv"' in cmd
+ assert 'export UV_PYTHON_INSTALL_DIR="/swebench_setup/python"' in cmd
+ assert 'export PATH="/swebench_setup/uv/bin:$PATH"' in cmd
+ # Dataset and prebuilt venv paths.
+ assert "--dataset_name /root/dataset/data.jsonl" in cmd
+ assert "/swebench_setup/SWE-bench/venv/bin/python" in cmd
+ assert "env -u VIRTUAL_ENV" in cmd
+ assert "cd /swebench_setup/SWE-bench " in cmd
+
+
+def test_run_eval_command_has_uv_exports_multilingual():
+ cmd = _eval_command(family="swe-bench-multilingual", metadata={"setup_dir": "/swebench_multilingual_setup"})
+ assert 'export UV_INSTALL_DIR="/swebench_multilingual_setup/uv"' in cmd
+ assert 'export UV_PYTHON_INSTALL_DIR="/swebench_multilingual_setup/python"' in cmd
+ assert 'export PATH="/swebench_multilingual_setup/uv/bin:$PATH"' in cmd
+ assert "/swebench_multilingual_setup/SWE-bench_Multilingual/venv/bin/python" in cmd
+
+
+def test_run_eval_command_uses_custom_setup_dir():
+ # When the verifier provisions a real host setup dir, the exports + venv path
+ # track it (uv venvs hardcode this absolute path).
+ cmd = _eval_command(family="swe-bench", metadata={"setup_dir": "/host/swe_swebench_setup"})
+ assert 'export UV_INSTALL_DIR="/host/swe_swebench_setup/uv"' in cmd
+ assert "/host/swe_swebench_setup/SWE-bench/venv/bin/python" in cmd
+
+
+# ---- mount source and eval cd/UV/venv path read ONE unified key -------------
+
+
+def _setup_srcs(spec) -> set[str]:
+ """Return the host source(s) of the setup-dir binds (every non-dataset mount).
+
+ Args:
+ spec: The sandbox spec whose ``provider_options["mounts"]`` is inspected.
+
+ Returns:
+ set[str]: The set of host source paths for non-dataset mounts.
+ """
+ return {m["src"] for m in spec.provider_options["mounts"] if m["dst"] != _DATASET_PATH}
+
+
+def test_setup_key_unified_via_host_setup_dir():
+ # build_spec mounts the host dir from the SAME key that run_eval reads for
+ # cd/UV/venv, so the prebuilt venv is bound and invoked from one path.
+ harness = SweBenchHarness("swe-bench")
+ meta = {"host_setup_dir": "/host/swe_swebench_setup"}
+ spec = harness.build_spec(_task(metadata=dict(meta)))
+ cmd = _eval_command(family="swe-bench", metadata=dict(meta))
+ # The bind source is exactly the path the eval command cd's into and runs
+ # the venv from.
+ assert _setup_srcs(spec) == {"/host/swe_swebench_setup"}
+ assert "cd /host/swe_swebench_setup/SWE-bench " in cmd
+ assert "/host/swe_swebench_setup/SWE-bench/venv/bin/python" in cmd
+ assert 'export UV_INSTALL_DIR="/host/swe_swebench_setup/uv"' in cmd
+
+
+def test_setup_key_unified_via_setup_dir():
+ # The same unified key resolves through the ``setup_dir`` alias too, so either
+ # config key keeps mount-source and eval-path in lockstep.
+ harness = SweBenchHarness("swe-bench")
+ meta = {"setup_dir": "/host/alt_setup"}
+ spec = harness.build_spec(_task(metadata=dict(meta)))
+ cmd = _eval_command(family="swe-bench", metadata=dict(meta))
+ assert _setup_srcs(spec) == {"/host/alt_setup"}
+ assert "cd /host/alt_setup/SWE-bench " in cmd
+ assert "/host/alt_setup/SWE-bench/venv/bin/python" in cmd
+
+
+def test_setup_key_default_alias_is_self_consistent():
+ # With NO host dir provisioned (SWE-bench-Verified default), both halves fall
+ # back to the family in-container alias, so the bind target == cd path and the
+ # venv resolves against the alias-mounted setup.
+ harness = SweBenchHarness("swe-bench")
+ spec = harness.build_spec(_task())
+ cmd = _eval_command(family="swe-bench")
+ assert _setup_srcs(spec) == {"/swebench_setup"}
+ assert "cd /swebench_setup/SWE-bench " in cmd
+ assert "/swebench_setup/SWE-bench/venv/bin/python" in cmd
+
+
+# ---- model_patch trailing-newline normalization in predictions -------------
+
+
+def _materialized_prediction(**task_overrides) -> dict:
+ """Materialize a task and return the parsed predictions record.
+
+ Args:
+ **task_overrides: Field values overriding the task defaults.
+
+ Returns:
+ dict: The single prediction record decoded from the uploaded JSONL.
+ """
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ async def run():
+ harness = SweBenchHarness("swe-bench")
+ task = _task(**task_overrides)
+ env = await AsyncSweEnvironment.start({"fake-swebench": {}}, harness.build_spec(task))
+ await harness.materialize(env, task)
+ return env.sandbox._provider.uploaded[_PREDICTIONS_PATH]
+
+ return json.loads(asyncio.run(run()))
+
+
+def test_materialize_normalizes_patch_trailing_newline():
+ # A non-empty patch missing its trailing newline gets one appended so the
+ # upstream ``git apply`` does not fail.
+ prediction = _materialized_prediction(model_patch="diff --git a/x b/x")
+ assert prediction["model_patch"] == "diff --git a/x b/x\n"
+
+
+def test_materialize_preserves_existing_trailing_newline():
+ # An already-terminated patch is left byte-for-byte unchanged (no double \n).
+ prediction = _materialized_prediction(model_patch="diff --git a/x b/x\n")
+ assert prediction["model_patch"] == "diff --git a/x b/x\n"
+
+
+def test_materialize_empty_patch_stays_empty():
+ # Only a truthy patch is normalized; an empty patch stays "" (it must not
+ # become a bare "\n", which would be a non-empty no-op patch to the grader).
+ prediction = _materialized_prediction(model_patch="")
+ assert prediction["model_patch"] == ""
+
+
+# ---- eval timeout threaded into env.execute ---------------------------------
+
+
+class _TimeoutRecordingProvider(_FakeProvider):
+ """Records the ``timeout_s`` passed to the eval exec call."""
+
+ name = "timeout-swebench"
+
+ def __init__(self, **kw):
+ super().__init__(**kw)
+ self.eval_timeout_s = None
+
+ async def exec(self, handle, command, *, cwd=None, env=None, timeout_s=None, user=None):
+ if "run_local_evaluation" in command:
+ self.eval_timeout_s = timeout_s
+ return await super().exec(handle, command, cwd=cwd, env=env, timeout_s=timeout_s, user=user)
+
+
+register_provider("timeout-swebench", _TimeoutRecordingProvider, override=True)
+
+
+def test_run_eval_threads_eval_timeout():
+ # run_eval must pass timeout_s = tests_timeout + 120 so a stuck nested harness
+ # is killed and masked rather than hanging the verifier.
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ async def run():
+ harness = SweBenchHarness("swe-bench")
+ task = _task(metadata={"tests_timeout": 600})
+ env = await AsyncSweEnvironment.start({"timeout-swebench": {}}, harness.build_spec(task))
+ await harness.materialize(env, task)
+ await harness.run_eval(env, task)
+ return env.sandbox._provider.eval_timeout_s
+
+ assert asyncio.run(run()) == 600 + 120
+
+
+def test_run_eval_threads_default_eval_timeout():
+ # With no explicit tests_timeout the default (1800) + 120 headroom is used.
+ from responses_api_agents.swe_env.environment import AsyncSweEnvironment
+
+ async def run():
+ harness = SweBenchHarness("swe-bench")
+ task = _task()
+ env = await AsyncSweEnvironment.start({"timeout-swebench": {}}, harness.build_spec(task))
+ await harness.materialize(env, task)
+ await harness.run_eval(env, task)
+ return env.sandbox._provider.eval_timeout_s
+
+ assert asyncio.run(run()) == 1800 + 120