Add initial repo template - #2
Merged
Merged
Conversation
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
pablo-garay
previously approved these changes
Aug 29, 2025
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
thomasdhc
approved these changes
Aug 29, 2025
abubakaria56
pushed a commit
to abubakaria56/Gym
that referenced
this pull request
Feb 28, 2026
…flagging Flagging validation issues and writing them into error.json
abubakaria56
pushed a commit
to abubakaria56/Gym
that referenced
this pull request
Mar 2, 2026
…flagging Flagging validation issues and writing them into error.json Signed-off-by: Dhrutisundar Sahoo <dhrutisundar.sahoo@turing.com>
abubakaria56
pushed a commit
to abubakaria56/Gym
that referenced
this pull request
Mar 2, 2026
…flagging Flagging validation issues and writing them into error.json Signed-off-by: Dhrutisundar Sahoo <dhrutisundar.sahoo@turing.com>
abubakaria56
pushed a commit
to abubakaria56/Gym
that referenced
this pull request
Mar 2, 2026
Add initial repo template
abubakaria56
pushed a commit
to abubakaria56/Gym
that referenced
this pull request
Mar 2, 2026
…flagging Flagging validation issues and writing them into error.json Signed-off-by: Dhrutisundar Sahoo <dhrutisundar.sahoo@turing.com>
abubakaria56
pushed a commit
to abubakaria56/Gym
that referenced
this pull request
Mar 2, 2026
Add initial repo template
abubakaria56
pushed a commit
to abubakaria56/Gym
that referenced
this pull request
Mar 2, 2026
…flagging Flagging validation issues and writing them into error.json Signed-off-by: Dhrutisundar Sahoo <dhrutisundar.sahoo@turing.com>
lbliii
added a commit
that referenced
this pull request
Apr 30, 2026
Three additions sourced from PR #1062 (skill-eval harness dogfooding). - Quickstart step 3: replace "Four Uvicorn lines print" with ng_status verification. Lawrence's flow uses ng_status; it's faster than scanning logs and produces an unambiguous "N healthy" signal. - Quickstart step 3: add a Tip for returning users — append +skip_venv_if_present=true to ng_run to skip venv re-creation. Real flag in cli_setup_command.py:116-117. - Configuration troubleshooting: new accordion for the trailing-slash gotcha (a "/" at the end of policy_base_url produces double-slash request paths some providers 404 on). From PR #1062 sharp edge #2. Verified ng_run blocks (cli.py:410 calls rh.run_forever); two-terminal pattern in Quickstart is correct, no change needed there. Signed-off-by: Lawrence Lane <llane@nvidia.com>
agronskiy
added a commit
that referenced
this pull request
May 20, 2026
…uting (#1367) ## Summary Two coupled changes that together fix walltime-induced rollout loss on multi-node GDPVal runs. **1. Per-task timeout** — default 3h30m, env `STIRRUP_PER_TASK_TIMEOUT_S` overrides. Wraps `await future` in `asyncio.wait_for`; on timeout, `ray.cancel(future, force=True)` + raises `TaskPerAttemptTimeoutError`. Logs once per process at first dispatch. **2. Failure classification + sidecar routing** — at the two `_build_failed_run_payload` callsites, every failure is classified into one of five classes and persisted accordingly: | class | persist | retry on chain-hop 2 | |---|---|---| | `kill_shaped` (Ray actor died, SIGTERM, OOM, node failure) | NO row anywhere | yes, unbounded (per-attempt timeout bounds wallclock) | | `timeout_exceeded` | 1 sidecar entry, `_ng_failure_terminal=True` | no | | `skipped` (TaskSampleSkipError) | 1 sidecar entry, terminal | no | | `transient` (verify-side 5xx, ConnectionError, asyncio.TimeoutError) | sidecar entry per attempt | yes, up to `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` (default 3) | | `legitimate` (real Python exception with user code in traceback) | sidecar entry per attempt | yes, up to max_attempts | Successes still write to `<output_jsonl_fpath>`; failures write to `<stem>_failures.jsonl`. `_load_from_cache` reads both: main jsonl is the success ledger, sidecar tracks attempts + terminal flags. The retry set is `materialized_inputs − (successes ∪ terminal ∪ maxed_out)`. ## Why this matters Without #1, a single pathological task that exceeds Slurm walltime can permanently consume every chain-hop's compute and never complete. Without #2, walltime-killed in-flight rollouts permanently disappear on chain-hop 2. The old `_load_from_cache` dedup keyed on `(task_index, rollout_index)` regardless of `-failed` status, so synthetic `-failed` rows written during the SIGTERM grace window looked already-done to the resumer. Worse, under harsh kills (SIGKILL, OOM) the `-failed` row write itself is non-atomic — we couldn't depend on it being there to filter. Using "row absent from main jsonl" as the canonical "needs retry" signal sidesteps both problems: kill_shaped writes nothing at all, so the disk state survives arbitrary kill timing. See the debug write-up for the production incident and design rationale: https://gitlab-master.nvidia.com/agronskiy/idea/-/blob/main/reports/debug/20260519T1011-gdpval-missing-histories.md ## Kill-shaped detection `_classify_rollout_failure` uses Ray's actor-died classes (`RayActorError`, `WorkerCrashedError`, `NodeDiedError`, `OutOfMemoryError`, `LocalRayletDiedError`) plus a user-code-in-traceback fallback for `RayTaskError`. The fallback distinguishes a real user exception (frames under \`responses_api_agents/\` or \`stirrup/\`) from Ray's internal post-mortem (e.g. summary-builder hitting a vanished worker log after Slurm's epilogue scrubbed `/tmp/ray`) — the latter is the walltime / SIGTERM signature and routes to `kill_shaped`. Detection fails open: if Ray's exception surface drifts, everything goes to bounded-retry `legitimate` instead of unbounded-retry `kill_shaped` — safe, not catastrophic. ## Knobs - `STIRRUP_PER_TASK_TIMEOUT_S` — per-attempt timeout (default 12600 s = 3h30m). - `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` — max retries per `(task_index, rollout_index)` (default 3). ## Test plan - \`python -m py_compile\` clean for both edited files. - ruff format clean. - Suggested smoke: run a GDPVal eval with `STIRRUP_PER_TASK_TIMEOUT_S=60`, confirm the log line is emitted once and that long-running rollouts get cancelled cleanly with `_ng_failure_class=timeout_exceeded` in `<stem>_failures.jsonl`. - Suggested integration: deliberately kill the deployment srun mid-run (`scancel -s TERM`), verify `_failures.jsonl` is unchanged (kill_shaped writes nothing) and that on chain-hop 2 the killed rollouts re-dispatch. --------- Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
wprazuch
added a commit
that referenced
this pull request
Jun 22, 2026
…M6g) Add a reusable LLM-as-judge resources-server template and a --template flag to ng_init_resources_server. `+template=judge` scaffolds a verifier whose verify() calls another model (judge / reward model / subagent) via the server client, wiring judge_model_server + judge_responses_create_params in the generated config, with a bounded judge-call semaphore and a clamped 0..1 reward parser. This is additive: the default 'basic' template is unchanged and the existing LLM-as-judge servers are left untouched (consolidating them is a separate, owner-sensitive change). New template files live under nemo_gym/resources/ and are omitted from coverage (copied verbatim into generated servers; exercised by the bundled test template inside a scaffolded server). Implements M6g (epic #1205, friction #2/#7). Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com>
This was referenced Jun 25, 2026
copy-pr-bot Bot
pushed a commit
that referenced
this pull request
Jun 25, 2026
…ool error Addresses review feedback on MCPResourcesServer: - (#3) Replace the per-process token->session dict with a stateless signed token (itsdangerous URLSafeSerializer keyed by the deterministic session-middleware secret). Any worker can verify a token another worker minted, so this works with num_workers > 1 and there is nothing to evict. - (#1) Offload blocking sync @gym_tool methods to a threadpool so they don't stall the event loop (and every concurrent rollout in the worker). - (#2) Raise a plain MCPSessionError with a clean message instead of HTTPException(401). MCP runs over JSON-RPC (HTTP 200), so the status code never reaches the client; FastMCP surfaces this as a tool error (isError: true). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>
mjmikulski
added a commit
that referenced
this pull request
Jul 16, 2026
…#2) - _sanitize_function_call_args: blanks the malformed function_call's arguments to "{}" in-place, matched by call_id, leaving good calls + tool outputs untouched (3 cases incl. no-match no-op). - responses(): a non-JSON tool-call argument does not crash the rollout; the /tool POST is skipped and the loop continues to a final answer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: MJ Mikulski <mmikulski@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add initial repo template