feat: harden SWE against filesystem overflow

[tdene]

apptainer's per-container session tmpfs has a size capped by sessiondir max size. SWE commonly exceeds the default value, with a simple git reset --hard usually killing the episode.

This PR backs the writable layer with a directory overlay that is freshly created under TMPDIR and removed when the container exits.

Episode results currently land next to the server sources and accumulate forever. This PR adds results_root to be able to specify a different root directory - and avoid disk overflow - and preserve_episode_artifacts which can clean up stale episode results if set to False.

[tdene]

/ok to test d68e7bc

[tdene]

/claude review

[claude]

RISK — overlay dirs leak on the exact failure mode they're meant to protect against (disk exhaustion).

trap 'rm -rf "$OVERLAY_DIR"' EXIT TERM INT only fires on those three signals. But this project's own teardown for timed-out / OOM episodes is SIGKILL, not SIGTERM: _kill_container_tree sends SIGKILL (app.py:2317, 2322) and _finish_container_command calls process.kill() (SIGKILL) on timeout (app.py:2494), as does _kill_active_command (app.py:2539). SIGKILL cannot be trapped, so the shell dies without running the trap and $OVERLAY_DIR under $TMPDIR is orphaned.

Blast radius: long training runs are precisely the ones that repeatedly hit agent/eval timeouts and the memory watchdog. Each killed episode leaves a full SWE working-tree overlay in $TMPDIR; these accumulate until $TMPDIR fills — the same "filesystem fills mid-run" failure preserve_episode_artifacts was added to prevent, but this path isn't governed by that knob and is never reaped.

Fix: track the overlay path Python-side and rmtree it in _kill_active_command / the timeout branch after the process is reaped, and/or add a startup sweep of stale apptainer_overlay_* dirs under $TMPDIR (the fixed prefix exists for exactly this). The trap alone only covers clean exits, not the kill paths that dominate at scale.

[claude]

SHIP WITH CARE

Sandbox-hardening + episode-scratch-lifecycle change to the SWE agent harness. Net improvement: the --writable-tmpfs → TMPDIR-backed directory overlay swap fixes real episode kills (working trees exceeding the capped tmpfs), the /etc/hosts and git safe.directory fixes are correctly made non-fatal, and preserve_episode_artifacts / results_root give operators a way to keep long runs from filling the shared FS. Tests cover the new paths well, including a stub-apptainer integration test for overlay lifecycle. No verifier/scoring or async-HTTP surface is touched — scores are unaffected.

One reliability finding worth resolving before this rides a long training run (inline on app.py:3362):

• RISK — overlay dirs leak under SIGKILL teardown. The overlay cleanup is a shell trap ... EXIT TERM INT, but the harness kills timed-out/OOM episodes with SIGKILL (_kill_container_tree, process.kill()), which is untrappable. Orphaned overlays accumulate in $TMPDIR until it fills — the same disk-exhaustion failure preserve_episode_artifacts guards against, but on a path that knob does not cover. Fix: reap the overlay Python-side in the kill/timeout branches and/or sweep stale apptainer_overlay_* at startup.

NOTE (author's call): results_root / preserve_episode_artifacts are new TypedDict-ish config fields with Python-side defaults on the Pydantic SWEBenchWrapperConfig. Per CLAUDE.md, defaults belong in the exemplar YAML — but the existing fields on this config (container_formatter, swebench_tests_timeout, etc.) already carry Python defaults, so this matches local convention rather than deviating from it. Not a blocker.

[claude]

RISK — overlay dirs leak on the exact path they're meant to protect against (disk exhaustion).

trap 'rm -rf "$OVERLAY_DIR"' EXIT TERM INT only fires on those three signals. But this project's own teardown for timed-out / OOM episodes is SIGKILL, not SIGTERM: _kill_container_tree sends SIGKILL (app.py:2317, 2322) and _finish_container_command calls process.kill() (SIGKILL) on timeout (app.py:2494), as does _kill_active_command (app.py:2539). SIGKILL cannot be trapped, so the shell dies without running the trap and $OVERLAY_DIR under $TMPDIR is orphaned.

Blast radius: long training runs are precisely the ones that hit agent/eval timeouts and the memory watchdog repeatedly. Each killed episode leaves a full SWE working-tree overlay in $TMPDIR. These accumulate until $TMPDIR fills — the same "filesystem fills mid-run" failure the preserve_episode_artifacts knob was added to prevent, but this path isn't governed by that knob and never gets reaped.

Fix: add a startup sweep of stale apptainer_overlay_* dirs under $TMPDIR (they carry a fixed prefix for exactly this reason), and/or add KILL handling by tracking the overlay path on the Python side and rmtree-ing it in _kill_active_command / the timeout branch after the process is reaped. The trap alone covers clean exits, not the kill paths that dominate at scale.

[tdene]

Thanks for the review, you're absolutely right. I've addressed this in my follow-up commit.

[tdene]

/ok to test 5eb218e

[tdene]

/claude review

[claude]

SHIP — LGTM, no reliability concerns.

Reviewed the SWE-bench harness sandbox-hardening + episode-cleanup change end to end:

• persistent_dir rmtree is correctly scoped (app.py:3585). It removes a per-episode directory ({instance_id}_{ts}_{uuid}, app.py:3447), not the shared results root. _inner_responses returns a fully-materialized NeMoGymResponse (metadata is in-memory JSON, no post-return file reads) before cleanup runs, and it's gated on not preserve_episode_artifacts — default True, so existing behavior is unchanged. Failed episodes take the except path and keep their dir + traceback.err, matching the documented contract.
• Overlay reap is env-consistent. Shell ${TMPDIR:-/tmp} and Python os.environ.get("TMPDIR") or "/tmp" resolve identically (both map unset/empty → /tmp), and both execute in the same runner_ray_remote process, so the reap targets the same path the launch shell created. The dirname is generated in Python and threaded onto ExecuteContainerCommandArgs/ActiveContainerCommand so the SIGKILL teardown paths (timeout branch of _finish_container_command, _kill_active_command) can reap by name where the shell's EXIT trap never fires.
• No leak on any exit path. The finally reap gates on returncode is not None; both normal completion and timeout-then-kill set it. Reap racing the shell's own trap is harmless (ignore_errors=True). Per-execution UUIDs prevent agent/eval overlay collisions.
• /etc/hosts and git safe.directory writes are made non-fatal — correct, these were a real episode-abort source under the overlay's uid remapping.
• Test coverage is strong: both kill paths parametrized, overlay lifecycle via stub apptainer, results_root relocation, and preserve_episode_artifacts both ways.

No public API, async, or scoring-correctness surface touched.