Add copy-pr-bot - #1
Merged
Merged
Conversation
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
pablo-garay
approved these changes
Aug 28, 2025
abubakaria56
pushed a commit
to abubakaria56/Gym
that referenced
this pull request
Mar 2, 2026
Add copy-pr-bot
abubakaria56
pushed a commit
to abubakaria56/Gym
that referenced
this pull request
Mar 2, 2026
Add copy-pr-bot
jsw-zorro
added a commit
to niletron/Gym
that referenced
this pull request
Apr 15, 2026
Added 3 tests covering gaps found in coverage audit:
- math: multi-part problem where correct answer is in non-final \boxed{}
(the NVIDIA-NeMo#1 math bug, 78% of 836 verified cases)
- mcqa: "Answer: X" format not caught in default mode (xfail, 205 cases)
- structured_outputs: <|im_end|> + <think> combined (xfail, 95% of failures)
Total: 49 tests (43 pass, 6 xfail documenting known bugs)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
jsw-zorro
added a commit
to niletron/Gym
that referenced
this pull request
Apr 15, 2026
structured_outputs: - Strip chat template tokens (<|im_end|>, <|im_start|>, etc.) before JSON parsing. vLLM/SGLang appends these during generation, causing json.loads() to fail with "Extra data" for valid JSON responses. This was the NVIDIA-NeMo#1 grading bug (100% failure rate in RLVR1 training). mcqa: - Strip <think>...</think> tags before answer extraction. re.search() was matching patterns inside think blocks instead of the actual answer outside, causing 91 false negatives in RLVR1 training. - Add "Answer: X" fallback pattern in default strict_single_letter_boxed mode. This format accounted for 205 of 431 MCQA grading bugs (48%). All 49 edge case tests now pass (0 xfail remaining). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
agronskiy
added a commit
that referenced
this pull request
May 20, 2026
…uting (#1367) ## Summary Two coupled changes that together fix walltime-induced rollout loss on multi-node GDPVal runs. **1. Per-task timeout** — default 3h30m, env `STIRRUP_PER_TASK_TIMEOUT_S` overrides. Wraps `await future` in `asyncio.wait_for`; on timeout, `ray.cancel(future, force=True)` + raises `TaskPerAttemptTimeoutError`. Logs once per process at first dispatch. **2. Failure classification + sidecar routing** — at the two `_build_failed_run_payload` callsites, every failure is classified into one of five classes and persisted accordingly: | class | persist | retry on chain-hop 2 | |---|---|---| | `kill_shaped` (Ray actor died, SIGTERM, OOM, node failure) | NO row anywhere | yes, unbounded (per-attempt timeout bounds wallclock) | | `timeout_exceeded` | 1 sidecar entry, `_ng_failure_terminal=True` | no | | `skipped` (TaskSampleSkipError) | 1 sidecar entry, terminal | no | | `transient` (verify-side 5xx, ConnectionError, asyncio.TimeoutError) | sidecar entry per attempt | yes, up to `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` (default 3) | | `legitimate` (real Python exception with user code in traceback) | sidecar entry per attempt | yes, up to max_attempts | Successes still write to `<output_jsonl_fpath>`; failures write to `<stem>_failures.jsonl`. `_load_from_cache` reads both: main jsonl is the success ledger, sidecar tracks attempts + terminal flags. The retry set is `materialized_inputs − (successes ∪ terminal ∪ maxed_out)`. ## Why this matters Without #1, a single pathological task that exceeds Slurm walltime can permanently consume every chain-hop's compute and never complete. Without #2, walltime-killed in-flight rollouts permanently disappear on chain-hop 2. The old `_load_from_cache` dedup keyed on `(task_index, rollout_index)` regardless of `-failed` status, so synthetic `-failed` rows written during the SIGTERM grace window looked already-done to the resumer. Worse, under harsh kills (SIGKILL, OOM) the `-failed` row write itself is non-atomic — we couldn't depend on it being there to filter. Using "row absent from main jsonl" as the canonical "needs retry" signal sidesteps both problems: kill_shaped writes nothing at all, so the disk state survives arbitrary kill timing. See the debug write-up for the production incident and design rationale: https://gitlab-master.nvidia.com/agronskiy/idea/-/blob/main/reports/debug/20260519T1011-gdpval-missing-histories.md ## Kill-shaped detection `_classify_rollout_failure` uses Ray's actor-died classes (`RayActorError`, `WorkerCrashedError`, `NodeDiedError`, `OutOfMemoryError`, `LocalRayletDiedError`) plus a user-code-in-traceback fallback for `RayTaskError`. The fallback distinguishes a real user exception (frames under \`responses_api_agents/\` or \`stirrup/\`) from Ray's internal post-mortem (e.g. summary-builder hitting a vanished worker log after Slurm's epilogue scrubbed `/tmp/ray`) — the latter is the walltime / SIGTERM signature and routes to `kill_shaped`. Detection fails open: if Ray's exception surface drifts, everything goes to bounded-retry `legitimate` instead of unbounded-retry `kill_shaped` — safe, not catastrophic. ## Knobs - `STIRRUP_PER_TASK_TIMEOUT_S` — per-attempt timeout (default 12600 s = 3h30m). - `NEMO_GYM_MAX_ROLLOUT_ATTEMPTS` — max retries per `(task_index, rollout_index)` (default 3). ## Test plan - \`python -m py_compile\` clean for both edited files. - ruff format clean. - Suggested smoke: run a GDPVal eval with `STIRRUP_PER_TASK_TIMEOUT_S=60`, confirm the log line is emitted once and that long-running rollouts get cancelled cleanly with `_ng_failure_class=timeout_exceeded` in `<stem>_failures.jsonl`. - Suggested integration: deliberately kill the deployment srun mid-run (`scancel -s TERM`), verify `_failures.jsonl` is unchanged (kill_shaped writes nothing) and that on chain-hop 2 the killed rollouts re-dispatch. --------- Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
8 tasks
This was referenced Jun 25, 2026
copy-pr-bot Bot
pushed a commit
that referenced
this pull request
Jun 25, 2026
…ool error Addresses review feedback on MCPResourcesServer: - (#3) Replace the per-process token->session dict with a stateless signed token (itsdangerous URLSafeSerializer keyed by the deterministic session-middleware secret). Any worker can verify a token another worker minted, so this works with num_workers > 1 and there is nothing to evict. - (#1) Offload blocking sync @gym_tool methods to a threadpool so they don't stall the event loop (and every concurrent rollout in the worker). - (#2) Raise a plain MCPSessionError with a clean message instead of HTTPException(401). MCP runs over JSON-RPC (HTTP 200), so the status code never reaches the client; FastMCP surfaces this as a tool error (isError: true). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Codex <codex@openai.com>
adil-a
added a commit
that referenced
this pull request
Jun 29, 2026
…ine) (#1802) Addresses @ffrujeri's review on #1572 (all 13 inline comments) plus the reward-profiling baseline. Targets `cmunley1/anyswe`. > Note: the swe_env `README.md` / `ng_test_all` CI-unblock landed separately via #1796 (merged). This PR is the review-comment fixes + docs + gold baseline on top of that. ## Review-comment fixes (@ffrujeri) | # | Comment | Fix | |---|---|---| | 1 | truncation mask never fires | inner agents (hermes/claude_code) now emit top-level `status="incomplete"` on internal max-turns/context/timeout; anyswe masks `resolved AND incomplete` (verified the status round-trips through `response.json`). openclaw exposes no internal signal → documented. | | 2 | openclaw config not runnable | select the apptainer provider for its `.sif` formatter; drop dead `apptainer_memory_limit_mb`/`skip_eval` fields. | | 3 | docker timeout gaps | configurable default `exec` timeout + bounded `close`/`cp` so a hung in-container command can't block a rollout. | | 4 | reward-profiling evidence | gold-patch baseline in the swe_env README (below). | | 5 | `_setup_params` assumes str `instance_dict` | accept str **or** dict (mirror `_build_swetask`). | | 6 | r2egym docstring | missing `eval_script` is unmasked reward-0, not an eval-error mask. | | 7 | missing docker binary | wrap `FileNotFoundError` → clear `SandboxCreateError`. | | 8 | unquoted shell interpolation | `shlex.quote` dataset values in `bash -c` (nv_internal/harness/swe_bench_ext). | | 9 | rc 125/126/127 infra-classify | narrow to **rc 125** only (126/127 are legit user-command codes). | | 10 | swe_rebench inflates on empty-required | empty-required guard (consistent with `compute_resolved`). | | 11 | unmasked infra failure on docker grade | mask `SandboxCreateError`/image-pull (`error_kind="sandbox"`). | | 12 | per-framework parsers untested | add `test_parsing_frameworks.py` (the 7 reviewer cases). | | 13 | docs discoverability | add a fern SWE-Environment page. | ## Gold-patch baseline (review #4) — at parity with the nested reference A full 500-instance SWE-bench Verified **gold-patch census on docker** resolves **493/500** (`patch_exists` 500/500, 0 infra errors), matching the apptainer/`.sif` *nested* reference **492/500** to within environment noise; empty patch **0/500**. (docker and apptainer both use the host-side flat grader here, so they're 1-1; the `.sif` figure is swebench's nested `run_evaluation`.) The census surfaced + fixed two real flat↔nested **reconstruction** gaps (445 → 486 → 493), verified with **0 regressions**: - **`PYTEST_ADDOPTS=-rA`**: swebench 4.1.0's eval for some families (sphinx via `tox`, several sklearn) runs pytest without `-rA`, so passing tests print only as dots and the host-side parser saw zero passes → ~45 unresolved even for gold (445→486). - **drop `GIT_CONFIG_GLOBAL=/dev/null`**: older images' git can't parse `/dev/null`, so the eval's `git checkout` + test-patch `git apply` failed → required tests "absent" (486→493). The remaining 7 misses are a small symmetric difference with `.sif` (4 shared genuine env-flaky: astropy-7606/8707/8872, django-10097; 3 docker-only sphinx instance-specific quirks; docker also resolves 4 that `.sif` misses). ## Validation - 258 unit tests pass (swe_env + docker provider + anyswe + claude_code/openclaw); ruff clean; `fern check` clean. - The `#1` truncation logic was adversarially verified end-to-end (top-level `status` survives the container→host round-trip; the mask is live). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Codex <codex@openai.com> Co-authored-by: Codex <codex@openai.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
waple0820
added a commit
to waple0820/Gym
that referenced
this pull request
Jul 23, 2026
…nd WebVoyager data bridge Make the environment reproducible end-to-end from one script so reviewers can verify it without guessing at the current CLI or the training recipe. - example.sh: idempotent, fail-fast, three explicit stages — Stage A (no GPU: backend test + gym serving stack + rollouts over data/example.jsonl against a policy endpoint), Stage B (1-GPU GRPO smoke via NeMo-RL), Stage C (same rollout on the Lexmount cloud backend via one flag). Policy endpoint supports both a generic OpenAI-compatible endpoint (POLICY_BASE_URL/API_KEY/MODEL) and a local vLLM model. - configs/grpo_lexmount_browser_smoke.yaml: 1-GPU GRPO smoke that ports the validated 0721 Qwen3-8B 2x8-NPU hyperparameters (GRPO group size 8, lr 5e-6 constant, 10 turns; reference reward 0.10 -> 0.29 over 60 steps). Every value is annotated as validated or smoke-scaled. - scripts/convert_webvoyager.py + data/webvoyager_sample.jsonl: map WebVoyager-style task JSON (MIT) into this env's example format, following the validated pipeline's cleaning conventions. Full 600-task set is not bundled; 3 verbatim sample tasks ship for --selftest. - README: how to serve the policy model (the NVIDIA-NeMo#1 reviewer stumbling block); clarify that the in-PR verify() is rule-based and the environment default, while the production-validated recipe uses a trajectory-level LLM judge; refresh the CLI to current main (openai_model policy, repo-relative paths). - data/example_rollouts.jsonl + example_metrics.json: 5 rollouts collected end-to-end against a Responses-API endpoint (reward 1.0 on the offline site). No base URL or key is stored in the committed artifacts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011FpVFa5xekvSpoAUkTFcrB Signed-off-by: waple0820 <feng.wang@lexmount.com>
waple0820
added a commit
to waple0820/Gym
that referenced
this pull request
Jul 23, 2026
…nd WebVoyager data bridge Make the environment reproducible end-to-end from one script so reviewers can verify it without guessing at the current CLI or the training recipe. - example.sh: idempotent, fail-fast, three explicit stages — Stage A (no GPU: backend test + gym serving stack + rollouts over data/example.jsonl against a policy endpoint), Stage B (1-GPU GRPO smoke via NeMo-RL), Stage C (same rollout on the Lexmount cloud backend via one flag). Policy endpoint supports both a generic OpenAI-compatible endpoint (POLICY_BASE_URL/API_KEY/MODEL) and a local vLLM model. - configs/grpo_lexmount_browser_smoke.yaml: 1-GPU GRPO smoke that ports the validated 0721 Qwen3-8B 2x8-NPU hyperparameters (GRPO group size 8, lr 5e-6 constant, 10 turns; reference reward 0.10 -> 0.29 over 60 steps). Every value is annotated as validated or smoke-scaled. - scripts/convert_webvoyager.py + data/webvoyager_sample.jsonl: map WebVoyager-style task JSON (MIT) into this env's example format, following the validated pipeline's cleaning conventions. Full 600-task set is not bundled; 3 verbatim sample tasks ship for --selftest. - README: how to serve the policy model (the NVIDIA-NeMo#1 reviewer stumbling block); clarify that the in-PR verify() is rule-based and the environment default, while the production-validated recipe uses a trajectory-level LLM judge; refresh the CLI to current main (openai_model policy, repo-relative paths). - data/example_rollouts.jsonl + example_metrics.json: 5 rollouts collected end-to-end against a Responses-API endpoint (reward 1.0 on the offline site). No base URL or key is stored in the committed artifacts. Signed-off-by: waple0820 <feng.wang@lexmount.com>
waple0820
added a commit
to waple0820/Gym
that referenced
this pull request
Jul 30, 2026
…nd WebVoyager data bridge Make the environment reproducible end-to-end from one script so reviewers can verify it without guessing at the current CLI or the training recipe. - example.sh: idempotent, fail-fast, three explicit stages — Stage A (no GPU: backend test + gym serving stack + rollouts over data/example.jsonl against a policy endpoint), Stage B (1-GPU GRPO smoke via NeMo-RL), Stage C (same rollout on the Lexmount cloud backend via one flag). Policy endpoint supports both a generic OpenAI-compatible endpoint (POLICY_BASE_URL/API_KEY/MODEL) and a local vLLM model. - configs/grpo_lexmount_browser_smoke.yaml: 1-GPU GRPO smoke that ports the validated 0721 Qwen3-8B 2x8-NPU hyperparameters (GRPO group size 8, lr 5e-6 constant, 10 turns; reference reward 0.10 -> 0.29 over 60 steps). Every value is annotated as validated or smoke-scaled. - scripts/convert_webvoyager.py + data/webvoyager_sample.jsonl: map WebVoyager-style task JSON (MIT) into this env's example format, following the validated pipeline's cleaning conventions. Full 600-task set is not bundled; 3 verbatim sample tasks ship for --selftest. - README: how to serve the policy model (the NVIDIA-NeMo#1 reviewer stumbling block); clarify that the in-PR verify() is rule-based and the environment default, while the production-validated recipe uses a trajectory-level LLM judge; refresh the CLI to current main (openai_model policy, repo-relative paths). - data/example_rollouts.jsonl + example_metrics.json: 5 rollouts collected end-to-end against a Responses-API endpoint (reward 1.0 on the offline site). No base URL or key is stored in the committed artifacts. Signed-off-by: waple0820 <feng.wang@lexmount.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add copy-pr-bot
They will just need cpu runners apparently. So should hopefully be able to just use the cpu machines from NV GH Runners for now.