Skip to content
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
0194e68
swe_env/anyswe: address PR #1572 review (A/B fixes + parsing tests)
codex Jun 27, 2026
d4bced2
docs: add SWE Environment page (swe_env + docker/apptainer provider)
codex Jun 27, 2026
13d660e
inner agents: emit Response.status=incomplete on truncation (activate…
codex Jun 27, 2026
9786e05
swe_env: force PYTEST_ADDOPTS=-rA in flat grading (fix sphinx/sklearn…
codex Jun 28, 2026
6434354
docs: add SWE-bench Verified gold-patch baseline to swe_env README
codex Jun 28, 2026
62102e0
swe_env: drop GIT_CONFIG_GLOBAL=/dev/null from eval env (fix git on o…
codex Jun 28, 2026
1a5019e
docs: update swe_env gold baseline to 493/500 (git-env fix -> .sif/ne…
codex Jun 28, 2026
4fcd2d6
swe_env: add gold_census.py CLI + reference from README
codex Jun 28, 2026
665d44f
anyswe: isolate apptainer eval from host $HOME (--no-mount home)
codex Jun 28, 2026
5359134
swe_env: give the flat eval command a provider-independent timeout (d…
codex Jun 28, 2026
abc7108
swe_env: make the eval-command timeout provider-independent everywhere
codex Jun 28, 2026
406728f
docs(swe_env): three-way gold-census reconciliation + correct repro
codex Jun 28, 2026
a563a8e
docs(swe_env): clarify apptainer-flat 491 vs nested .sif 492 vs flat 493
codex Jun 28, 2026
2dd9ad1
gold_census: on-demand apptainer sif build + --tests-timeout for exac…
codex Jun 28, 2026
3c06623
docs(swe_env): verified EXACT docker==apptainer flat parity (493/500 …
codex Jun 28, 2026
91155fe
docs: correct stale apptainer .sif 492 baseline -> verified flat 493=…
codex Jun 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 69 additions & 0 deletions fern/versions/latest/pages/agent-server/swe-environment.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
---
title: "SWE Environment"
description: "Run any Gym agent inside a SWE-bench task container and grade its patch, on docker or apptainer."
position: 3
---

# SWE Environment

The **SWE environment** lets you evaluate any Gym agent on software-engineering benchmarks
(SWE-bench Verified / Multilingual, SWE-bench-ext, R2E-Gym, SWE-rebench) by dropping the agent
*inside* the task's container, extracting its `git diff`, and grading that patch against the
instance's tests. It is harness-agnostic (the same flow runs hermes, claude_code, or openclaw)
and sandbox-agnostic (docker or apptainer).

It has two pieces:

- **`responses_api_agents/anyswe_agent`** — the thin runner (an Agent Server). It provisions one
sandbox from the task image, runs the configured inner agent inside it via a generic
dynamic-loader runner, extracts the patch (`git add -A && git diff --cached`), and grades it.
- **`responses_api_agents/swe_env`** — the shared library the runner builds on (not a server):
- **`harnesses/`** — per-dataset-family recipes: build the sandbox spec, materialize the patch,
run the evaluation, and grade the log host-side with the official per-repo parser.
- **`sandbox.py`** — async sandbox lifecycle (`AsyncSweEnvironment`, `acquire_sandbox`) over a
`nemo_gym.sandbox` provider, with always-teardown.
- **`self_drive.py`** — provision a writable sandbox, inject a sandbox-reachable model endpoint,
run the agent, and extract the diff.
- **`verify_task.py`** — grade a patch **inline** in a fresh sandbox (no separate verifier
server), returning a mask-aware reward.

## Selecting the sandbox provider

The same task runs on either backend via `sandbox_provider`. **docker** pulls the per-instance
image on demand (no pre-build); **apptainer** runs a pre-built local `.sif`.

```yaml
# docker (default): images pull on demand from the registry
anyswe_hermes:
responses_api_agents:
anyswe_agent:
container_formatter: "docker://swebench/sweb.eval.x86_64.{instance_id}"
sandbox_provider: {docker: {}}

# apptainer: point the formatter at a local .sif and select the provider
container_formatter: "data/sifs/{instance_id}.sif"
sandbox_provider: {apptainer: {}}
```

<Note>
Grading is host-side (flat): the eval log is parsed with swebench's official per-repo parser, so
it runs on any exec-capable provider — docker, apptainer, or opensandbox. The two backends produce
equivalent resolve rates; see the in-tree `responses_api_agents/swe_env/README.md` for the
gold-patch baseline.
</Note>

## Running it

```bash
# Prepare a dataset of tasks (optionally pre-build .sif images for apptainer)
python responses_api_agents/anyswe_agent/prepare.py --limit 100

# Start the servers, then collect rollouts
ng_run "+config_paths=[responses_api_agents/anyswe_agent/configs/anyswe_hermes.yaml,responses_api_models/openai_model/configs/openai_model.yaml]"
ng_collect_rollouts +agent_name=anyswe_hermes \
+input_jsonl_fpath=responses_api_agents/anyswe_agent/data/swebench_verified.jsonl \
+output_jsonl_fpath=results/hermes.jsonl +num_samples_in_parallel=16
```

The resolve rate is `mean/reward` in the run's `aggregate_metrics.json`; a clean run has
`mean/mask_sample == 0` (infra failures and timeouts are masked out, not scored 0).
57 changes: 40 additions & 17 deletions nemo_gym/sandbox/providers/docker/provider.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ def __init__(
run_args: list[str] | None = None,
keep_alive_command: str = "sleep infinity",
concurrency: int = 32,
default_exec_timeout_s: int | float | None = 3600,
**_: Any,
) -> None:
"""Configure the Docker sandbox provider.
Expand All @@ -70,6 +71,9 @@ def __init__(
it alive for subsequent ``exec`` calls.
concurrency: Maximum number of concurrent ``docker`` CLI subprocesses,
bounded by a shared semaphore (matches the apptainer provider).
default_exec_timeout_s: Default per-``exec`` timeout (seconds) applied when a
caller passes none, so a hung in-container command cannot block a rollout
forever (e.g. the git-diff extraction in self_drive). None = no default.
**_: Additional keyword arguments are accepted and ignored.

Raises:
Expand All @@ -83,6 +87,7 @@ def __init__(
self._run_args = list(run_args or [])
self._keep_alive = keep_alive_command
self._semaphore = asyncio.Semaphore(concurrency)
self._default_exec_timeout_s = default_exec_timeout_s

async def _run(self, *args: str, timeout_s: int | float | None = None) -> tuple[int, str, str]:
"""Run the ``docker`` CLI with the given arguments and capture output.
Expand All @@ -101,12 +106,19 @@ async def _run(self, *args: str, timeout_s: int | float | None = None) -> tuple[
text using ``errors="replace"``.
"""
async with self._semaphore:
proc = await asyncio.create_subprocess_exec(
self._bin,
*args,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
try:
proc = await asyncio.create_subprocess_exec(
self._bin,
*args,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
except FileNotFoundError as exc:
# Surface a clear, actionable error instead of a bare FileNotFoundError deep in
# create()/exec() (parity with apptainer's _require_apptainer up-front check).
raise SandboxCreateError(
f"docker executable {self._bin!r} not found on PATH — install Docker or set docker_bin"
) from exc
try:
out, err = await asyncio.wait_for(proc.communicate(), timeout=timeout_s)
except (asyncio.TimeoutError, TimeoutError):
Expand Down Expand Up @@ -228,15 +240,16 @@ async def exec(
cwd: Working directory for the command; falls back to the workdir
recorded at create time.
env: Extra environment variables for the command.
timeout_s: Optional timeout in seconds; on expiry a result with
return code 124 and ``error_type="timeout"`` is returned.
timeout_s: Optional timeout in seconds; falls back to the provider's
``default_exec_timeout_s`` when None. On expiry a result with return
code 124 and ``error_type="timeout"`` is returned.
user: User (name or UID) to run as; falls back to the provider's
default user.

Returns:
A ``SandboxExecResult`` with stdout, stderr, return code, and an
``error_type`` of ``"sandbox"`` for docker-level failures (125/126/
127 with no stdout), ``"timeout"`` on timeout, or None otherwise.
``error_type`` of ``"sandbox"`` for a docker-daemon failure (rc 125 with
no stdout), ``"timeout"`` on timeout, or None otherwise.
"""
args = ["exec"]
workdir = cwd or handle.raw.get("workdir")
Expand All @@ -248,17 +261,22 @@ async def exec(
for key, value in (env or {}).items():
args += ["-e", f"{key}={value}"]
args += [handle.sandbox_id, "bash", "-c", command]
eff_timeout = timeout_s if timeout_s is not None else self._default_exec_timeout_s
try:
rc, out, err = await self._run(*args, timeout_s=timeout_s)
rc, out, err = await self._run(*args, timeout_s=eff_timeout)
except (asyncio.TimeoutError, TimeoutError):
# Only the local `docker exec` client is killed here; the in-container process is
# reaped when the sandbox is closed (`docker rm -f`), which acquire_sandbox always does.
return SandboxExecResult(
stdout=None,
stderr=f"command timed out after {timeout_s}s",
stderr=f"command timed out after {eff_timeout}s",
return_code=124,
error_type="timeout",
)
# docker exec returns 125/126/127 for docker-level failures (container gone, not executable).
error_type = "sandbox" if rc in (125, 126, 127) and not out else None
# rc 125 is a docker-daemon-level failure (container gone / daemon error). 126 (not
# executable) and 127 (command not found) are legitimate *user*-command exit codes when
# run via `bash -c`, so only rc 125 with no stdout is classified as an infra failure.
error_type = "sandbox" if rc == 125 and not out else None
return SandboxExecResult(stdout=out, stderr=err, return_code=rc, error_type=error_type)

async def upload_file(self, handle: SandboxHandle, source_path: Path, target_path: str) -> None:
Expand All @@ -275,7 +293,7 @@ async def upload_file(self, handle: SandboxHandle, source_path: Path, target_pat
parent = posixpath.dirname(target_path)
if parent:
await self.exec(handle, f"mkdir -p {shlex.quote(parent)}")
rc, out, err = await self._run("cp", str(source_path), f"{handle.sandbox_id}:{target_path}")
rc, out, err = await self._run("cp", str(source_path), f"{handle.sandbox_id}:{target_path}", timeout_s=300)
if rc != 0:
raise RuntimeError(f"docker cp upload failed: {err.strip() or out.strip()}")

Expand All @@ -292,7 +310,7 @@ async def download_file(self, handle: SandboxHandle, source_path: str, target_pa
"""
target = Path(target_path)
target.parent.mkdir(parents=True, exist_ok=True)
rc, out, err = await self._run("cp", f"{handle.sandbox_id}:{source_path}", str(target))
rc, out, err = await self._run("cp", f"{handle.sandbox_id}:{source_path}", str(target), timeout_s=300)
if rc != 0:
raise RuntimeError(f"docker cp download failed: {err.strip() or out.strip()}")

Expand All @@ -317,7 +335,12 @@ async def close(self, handle: SandboxHandle) -> None:
Args:
handle: Handle identifying the container to remove.
"""
await self._run("rm", "-f", handle.sandbox_id)
# Best-effort + bounded: teardown runs in acquire_sandbox's finally, so a wedged daemon
# must not hang (or crash) it after the result is already in hand.
try:
await self._run("rm", "-f", handle.sandbox_id, timeout_s=120)
except Exception:
pass

async def aclose(self) -> None:
"""Release provider-level resources; this provider holds none."""
Expand Down
7 changes: 6 additions & 1 deletion responses_api_agents/anyswe_agent/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -447,7 +447,12 @@ def _setup_params(self, body: NeMoGymResponseCreateParamsNonStreaming) -> Tuple[
dataset_dir = persistent_dir / "instance_datasets"
dataset_dir.mkdir(parents=True, exist_ok=True)
instance_dataset_path = dataset_dir / f"{agent_run_id}.jsonl"
instance_dict = json.loads(problem_info["instance_dict"])
# Accept instance_dict as a JSON string or an already-parsed dict (mirrors _build_swetask),
# so a dict-valued row doesn't raise TypeError and mask the whole run with an opaque error.
raw_instance_dict = problem_info["instance_dict"]
instance_dict = (
json.loads(raw_instance_dict) if isinstance(raw_instance_dict, str) else dict(raw_instance_dict)
)
instance_dict.setdefault("repo_name", instance_dict.get("repo", ""))
instance_dataset_path.write_text(json.dumps(instance_dict) + "\n")

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,11 +42,10 @@ anyswe_openclaw:
api: openai-completions

container_formatter: data/sifs/{instance_id}.sif
sandbox_provider: {apptainer: {}}
swebench_agent_timeout: 2100
swebench_tests_timeout: 1800
apptainer_memory_limit_mb: 32768
concurrency: 2
skip_eval: false

datasets:
- name: example
Expand Down
17 changes: 15 additions & 2 deletions responses_api_agents/claude_code_agent/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ def parse_stream_json(stdout: str) -> tuple[list[Any], dict]:
total_input = 0
total_output = 0
num_turns: Optional[int] = None
incomplete = False

for event in raw_events:
etype = event.get("type")
Expand All @@ -100,6 +101,13 @@ def parse_stream_json(stdout: str) -> tuple[list[Any], dict]:
# Claude Code's authoritative turn counter (what --max-turns bounds).
if event.get("num_turns") is not None:
num_turns = int(event["num_turns"])
# The result subtype distinguishes a clean finish ("success") from an internal cutoff
# ("error_max_turns") or a mid-run error ("error_during_execution"); a non-success
# result means the agent did not finish cleanly. (Subprocess timeouts are masked
# upstream by anyswe's agent_timed_out, separately from this.)
subtype = event.get("subtype")
if event.get("is_error") or (subtype is not None and subtype != "success"):
incomplete = True

elif etype == "assistant":
message = event.get("message", {})
Expand Down Expand Up @@ -174,7 +182,7 @@ def parse_stream_json(stdout: str) -> tuple[list[Any], dict]:
)
)

metadata: dict = {"input_tokens": total_input, "output_tokens": total_output}
metadata: dict = {"input_tokens": total_input, "output_tokens": total_output, "incomplete": incomplete}
if num_turns is not None:
metadata["num_turns"] = num_turns
return output_items, metadata
Expand Down Expand Up @@ -388,7 +396,11 @@ async def _run_claude_code(
proc.kill()
await proc.communicate()
LOG.warning("claude-code timed out after %ds", self.config.timeout)
return "", model
# An internal timeout produces no real `result` event; emit a synthetic one so
# parse_stream_json marks the response incomplete. This truncation can fire before
# anyswe's outer agent timeout (inner timeout < swebench_agent_timeout), so the
# agent must self-report it rather than relying on the outer agent_timed_out guard.
return json.dumps({"type": "result", "subtype": "error_timeout", "is_error": True}) + "\n", model

if proc.returncode not in (0, None):
LOG.warning("claude-code exited %d: %s", proc.returncode, stderr.decode(errors="replace")[:500])
Expand Down Expand Up @@ -502,6 +514,7 @@ async def _create_response(
model=model_name,
object="response",
output=output_items,
status="incomplete" if usage.get("incomplete") else "completed",
tool_choice=body.tool_choice,
tools=body.tools,
parallel_tool_calls=body.parallel_tool_calls,
Expand Down
21 changes: 18 additions & 3 deletions responses_api_agents/claude_code_agent/tests/test_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ async def fake_exec(*cmd, **kwargs):
assert "result" in stdout
assert model == "claude-sonnet-4-6"

def test_timeout_returns_empty(self, tmp_path: Path) -> None:
def test_timeout_emits_incomplete_result(self, tmp_path: Path) -> None:
agent = _make_agent(timeout=1)
killed = {"called": False}

Expand All @@ -249,9 +249,12 @@ async def fake_wait_for(coro, timeout):
):
stdout, model = asyncio.run(agent._run_claude_code("hello"))

assert stdout == ""
assert killed["called"] is True
assert model == "claude-sonnet-4-6"
# An internal timeout self-reports as an incomplete run via a synthetic result event, so a
# truncation that fires before anyswe's outer agent timeout is still masked downstream.
_, usage = parse_stream_json(stdout)
assert usage["incomplete"] is True


class TestRolloutMCPConfig:
Expand Down Expand Up @@ -470,7 +473,19 @@ def _user_tool_result(self, tool_use_id: str, result: str) -> str:
def test_empty(self) -> None:
items, usage = parse_stream_json("")
assert items == []
assert usage == {"input_tokens": 0, "output_tokens": 0}
assert usage == {"input_tokens": 0, "output_tokens": 0, "incomplete": False}

def test_result_success_is_not_incomplete(self) -> None:
line = _event("result", subtype="success", is_error=False, num_turns=3)
_, usage = parse_stream_json(line)
assert usage["incomplete"] is False

def test_result_error_subtype_marks_incomplete(self) -> None:
# An internal cutoff (max-turns) or mid-run error means the agent did not finish cleanly.
for sub in ("error_max_turns", "error_during_execution"):
line = _event("result", subtype=sub, is_error=True, num_turns=90)
_, usage = parse_stream_json(line)
assert usage["incomplete"] is True, sub

def test_text_message(self) -> None:
line = self._assistant([{"type": "text", "text": "hello"}])
Expand Down
5 changes: 5 additions & 0 deletions responses_api_agents/hermes_agent/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -359,12 +359,17 @@ def _patched_build_api_kwargs(api_messages):
)
)

# The agent ran out of turns / blew its context window if run_conversation reports it did
# not complete (api_call_count >= max_iterations). Mark the response incomplete so a
# downstream consumer (e.g. anyswe) can mask an accidental pass instead of scoring it 1.0.
agent_completed = bool(result.get("completed", True))
return NeMoGymResponse(
id=f"resp_{uuid4().hex}",
created_at=int(time()),
model=model_name,
object="response",
output=output_items,
status="completed" if agent_completed else "incomplete",
tool_choice=body.tool_choice,
tools=body.tools,
parallel_tool_calls=body.parallel_tool_calls,
Expand Down
5 changes: 5 additions & 0 deletions responses_api_agents/openclaw_agent/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,11 @@ async def responses(
input_tokens = usage.get("input_tokens", 0)
output_tokens = usage.get("output_tokens", 0)

# OpenClaw's output envelope exposes no internal truncation / stop-reason signal (only the
# final text + usage), so — unlike hermes and claude_code — we cannot mark the response
# "incomplete" on an internal max-turns / context cutoff. Subprocess-timeout truncation is
# the observable case and is masked upstream by anyswe's agent_timed_out. If OpenClaw later
# surfaces a stop reason, set status="incomplete" here.
return NeMoGymResponse(
id=f"resp_{uuid4().hex}",
created_at=int(time()),
Expand Down
Loading
Loading