feat(sandbox): ECS Fargate sandbox provider - #2
Closed
Glorf wants to merge 12 commits into
Closed
Conversation
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
Port NEL's ECS Fargate sandboxing into Gym as a third sandbox provider, stacked on the provider framework from NVIDIA-NeMo#1377. - nemo_gym/sandbox/providers/ecs_fargate/{engine,provider}.py — NEL's engine lifted (task-def registration + SSM caching, RunTask with capacity retries, SSH sidecar + reverse-tunnel outside-endpoint routing, in-container exec server, CodeBuild->ECR image build) behind a thin adapter that keeps per-sandbox state in SandboxHandle.raw. - registry loader + `sandbox-ecs` extra (boto3, lazy-imported). - region-only config via SSM autodiscovery (/<ssm_project>/ecs-sandbox/config), matching NEL. - mini_swe_agent_2 ecs_fargate config + 14 unit tests (AWS/SSH mocked). Fast-follow P0 (separate PR): route the model endpoint over Teleport as an internal option instead of the globally-exposed SSH reverse tunnel that security flagged; SSH tunneling stays available for community use. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf
changed the title
- ECS Fargate provider mirrors a missing public image into the ECR mirror on demand during create (ImageBuilder.ensure_mirrored, gated by the new auto_mirror config, default on), with the same in-flight dedup as the environment_dir build path. Callers no longer pre-stage images. - Fix MiniSWESandboxEnvironment conda activation: source conda from known install roots instead of `conda info --base`, which fails in the ECS exec server's non-login shell and was silently dropping the golden gold-patch apply (zeroing reward). - Add the sandbox-ecs extra to mini_swe_agent_2 requirements so the agent's per-server venv can run the ECS provider under ng_run. - Docs: ECS Fargate provider README (general, provider-scoped) + a concise SWE-bench-on-ECS run guide in the agent README. - Tests for mirror resolution/dedup/auto-mirror hook and conda command build. Signed-off-by: Michal Bien <mbien@nvidia.com>
Glorf
changed the title
hemildesai
pushed a commit
that referenced
this pull request
Jun 9, 2026
…uting (NVIDIA-NeMo#1367) ## Summary Two coupled changes that together fix walltime-induced rollout loss on multi-node GDPVal runs. **1. Per-task timeout** — default 3h30m, env `STIRRUP_PER_TASK_TIMEOUT_S` overrides. Wraps `await future` in `asyncio.wait_for`; on timeout, `ray.cancel(future, force=True)` + raises `TaskPerAttemptTimeoutError`. Logs once per process at first dispatch. **2. Failure classification + sidecar routing** — at the two `_build_failed_run_payload` callsites, every failure is classified into one of five classes and persisted accordingly: | class | persist | retry on chain-hop 2 | |---|---|---| | `kill_shaped` (Ray actor died, SIGTERM, OOM, node failure) | NO row anywhere | yes, unbounded (per-attempt timeout bounds wallclock) | | `timeout_exceeded` | 1 sidecar entry, `_ng_failure_terminal=True` | no | | `skipped` (TaskSampleSkipError) | 1 sidecar entry, terminal | no | | `transient` (verify-side 5xx, ConnectionError, asyncio.TimeoutError) | sidecar entry per attempt | yes, up to `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` (default 3) | | `legitimate` (real Python exception with user code in traceback) | sidecar entry per attempt | yes, up to max_attempts | Successes still write to `<output_jsonl_fpath>`; failures write to `<stem>_failures.jsonl`. `_load_from_cache` reads both: main jsonl is the success ledger, sidecar tracks attempts + terminal flags. The retry set is `materialized_inputs − (successes ∪ terminal ∪ maxed_out)`. ## Why this matters Without #1, a single pathological task that exceeds Slurm walltime can permanently consume every chain-hop's compute and never complete. Without #2, walltime-killed in-flight rollouts permanently disappear on chain-hop 2. The old `_load_from_cache` dedup keyed on `(task_index, rollout_index)` regardless of `-failed` status, so synthetic `-failed` rows written during the SIGTERM grace window looked already-done to the resumer. Worse, under harsh kills (SIGKILL, OOM) the `-failed` row write itself is non-atomic — we couldn't depend on it being there to filter. Using "row absent from main jsonl" as the canonical "needs retry" signal sidesteps both problems: kill_shaped writes nothing at all, so the disk state survives arbitrary kill timing. See the debug write-up for the production incident and design rationale: https://gitlab-master.nvidia.com/agronskiy/idea/-/blob/main/reports/debug/20260519T1011-gdpval-missing-histories.md ## Kill-shaped detection `_classify_rollout_failure` uses Ray's actor-died classes (`RayActorError`, `WorkerCrashedError`, `NodeDiedError`, `OutOfMemoryError`, `LocalRayletDiedError`) plus a user-code-in-traceback fallback for `RayTaskError`. The fallback distinguishes a real user exception (frames under \`responses_api_agents/\` or \`stirrup/\`) from Ray's internal post-mortem (e.g. summary-builder hitting a vanished worker log after Slurm's epilogue scrubbed `/tmp/ray`) — the latter is the walltime / SIGTERM signature and routes to `kill_shaped`. Detection fails open: if Ray's exception surface drifts, everything goes to bounded-retry `legitimate` instead of unbounded-retry `kill_shaped` — safe, not catastrophic. ## Knobs - `STIRRUP_PER_TASK_TIMEOUT_S` — per-attempt timeout (default 12600 s = 3h30m). - `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` — max retries per `(task_index, rollout_index)` (default 3). ## Test plan - \`python -m py_compile\` clean for both edited files. - ruff format clean. - Suggested smoke: run a GDPVal eval with `STIRRUP_PER_TASK_TIMEOUT_S=60`, confirm the log line is emitted once and that long-running rollouts get cancelled cleanly with `_ng_failure_class=timeout_exceeded` in `<stem>_failures.jsonl`. - Suggested integration: deliberately kill the deployment srun mid-run (`scancel -s TERM`), verify `_failures.jsonl` is unchanged (kill_shaped writes nothing) and that on chain-hop 2 the killed rollouts re-dispatch. --------- Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
hemildesai
force-pushed
the
hemil/sandbox-api-part-1
branch
3 times, most recently
from
June 11, 2026 21:36
8127481 to
428238f
Compare
This was referenced Jun 16, 2026
Author
|
Superseded by NVIDIA-NeMo#1645 (ECS provider, now in the upstream GitHub stack on sandbox-api-part-1). |
Author
|
Reopened — closed in error, please disregard the previous comment. Leaving this PR as-is. |
Author
|
Re-closing: superseded by NVIDIA-NeMo#1645 (ECS provider, now in the upstream GitHub stack on sandbox-api-part-1). My earlier reopen was just me double-checking PR ownership. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port NEL's ECS Fargate sandboxing into Gym as a third sandbox provider, stacked on the provider framework from NVIDIA-NeMo#1377.
sandbox-ecsextra (boto3, lazy-imported).Fast-follow P0 (separate PR): route the model endpoint over Teleport as an internal option instead of the globally-exposed SSH reverse tunnel that security flagged; SSH tunneling stays available for community use.